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FOREWORD 


This  report  may  be  of  most  value  to  the  relatively  inexperienced  human  factors  engineer 
who  must  support  the  analysis  activities  associated  with  a  systems  design  effort.  The  human 
factors  engineer  can  contribute  a  lot  to  such  an  effort,  and  insure  that  sufficient  attention  is 
paid  to  the  operator  performance  part  of  the  analysis.  With  practice,  the  human  factors 
engineer  can  do  most  of  the  analysis,  providing  useful  results  to  the  rest  of  the  design  team;  the 
results  can  also  be  used  to  support  human  operator  requirements  with  hard  numbers. 

Systems  engineers  may  also  gain  an  understanding  from  this  report  of  how  operator 
performance  considerations  can  be  included  in  systems  analysis  and  design. 

The  report  outlines  the  procedures  to  follow  in  conducting  a  systems  effectiveness  analysis. 
The  components  of  such  an  analysis,  including  operator  performance  data  are  described  and 
examples  are  provided.  Although  the  report  does  not  provide  specific  instruction  on  how  to  do 
a  task  analysis  or  function  allocation,  or  how  to  collect  performance  data,  the  analytic 
framework  is  provided  for  combining  these  inputs  in  an  effectiveness  analysis. 
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INTRODUCTION 


This  report  develops  the  procedures  to  be  followed  to  predict  the  ability  of  a 
human  operator  to  use  a  system  under  real  world  conditions.  Can  a  bus  driver 
complete  his  route  on  time,  winter  or  summer?  Can  a  pilot  find  a  target  in  time 
to  attack  it,  in  the  desert  or  in  mountains? 

The  end  product  of  such  an  analysis  is  the  probability  of  success,  or  the 
percent  of  the  time  that  the  system  could  be  used  successfully.  Such 
quantitative  results  can  provide  the  designers  of  new  equipment  with  guidelines 
as  to  how  good  or  poor  their  design  is;  the  results  can  illustrate  the  importance 
of- including  human  engineering  in  the  design  process. 

Human  Factors  specialists  have  participated  in  the  design  of  systems  for 
over  thirty  years.  The  level  of  this  participation  has  varied  from  determining 
the  shape  of  knobs  on  a  control  panel  to  developing  employment  concepts  that 
directly  affect  much  broader  design  questions.  The  work  has  ranged  from 
designing  and  conducting  laboratory  experiments  and  man-in-the-loop 
simulations  to  paper  analyses  of  tasks  and  operational  sequences. 

The  application  of  this  broad  range  of  human  factors  specialties  to  major 
design  efforts,  such  as  the  Navy's  F/A-18  aircraft  has  illustrated  the  acceptance 
of  human  factors  work  in  the  design  process.  That  acceptance  is  not  universal, 
however,  and  human  factors  people  continue  to  worry  about  being  included  on 
the  design  team.  They  worry  about  advancing  their  specialty  to  keep  up  with  the 
changing  technical  world,  and  often  must  "prove  the  worth"  of  human  factors 
contributions  to  system  design  efforts. 

In  a  survey  of  senior  level  human  factors  specialists  conducted  by  Meister,* 
64%  felt  that  designers  on  their  own  are  incapable  of  understanding  human 

factors  inputs.  In  a  later  survey  by  Meister,  "if  new  methodological  approaches 
are  needed,  no  one  seems  to  know  what  these  would  be.”  One  of  the  main 
problems  cited  by  the  respondents  was  that  of  demonstrating  the  worth  of  human 
factors.  It  should  be  noted  that  the  surveys  did  not  ask  systems  analysts  or 
design  engineers  about  human  factors  or  the  inclusion  of  operator  performance  in 
their  work.  This  should  be  done  to  get  a  more  complete  description  of  the 
situation. 

1  Meister,  David,  "The  Influence  of  Government  on  Human  Factors  Research 

and  Development,"  Proceedings.  1979  Human  Factors  Society  Annual  Meeting  (pp. 
5-13).  ” 

2 

Meister,  David,  "The  Present  and  Future  of  Human  Factors,"  Applied 
Ergonomics  1982,  Vol.  13.4,  pp.  281-287. 

3 


NWC  TP  6541 


3 

In  another  such  survey  conducted  by  Topmiller,  et  al  ,  as  part  of  a 
methodological  study,  the  authors  concluded  that: 


"It  is  fairly  obvious  that  most  of  the  advanced  thinkers  in 
the  human  factors  discipline  believe  that  the  greatest  needs 
for  future  technology  development  are  being  driven  by  the 
requirement  for  a  human-machine-mission  (H-M-M)  systems 
analytic  and  simulation  capability.  H-M-M  systems  analysis 
and  simulation  methods  must  be  developed  to  treat  human, 
equipment  and  mission  parameters  in  equivalent  quantitative 
terms  in  order  to  isolate  this  respective  contribution  to  overall 


systems  effectiveness. 


II 


Perhaps  the  human  factors  people  must  move  further  into  the  design  process 
with  their  data,  and  do  something  with  it  themselves.  This  report  will  help  them 
do  that. 


SCOPE 

Before  system  performance  can  be  predicted,  it  must  be  defined.  According 
to  a  USAF  Weapons  System  Effectiveness  Industry  Advisory  Committee  (as  cited 
by  Kline), ^ 

"System  Effectiveness  is  a  measure  of  the  extent  to  which 
a  system  may  be  expected  to  achieve  a  set  of  specific  mission 
requirements  and  is  a  function  of  availability,  dependability, 
and  capability." 

This  report  does  three  things: 

1.  Gives  a  very  brief  overview  of  current  analysis  methods  used  in  the 
estimation  of  the  capability  part  of  the  effectiveness  equation. 

2.  Discusses  the  factors  that  must  be  included  in  systems 
effectiveness  computations. 


^  Topmiller,  Donald  A.,  Methods:  Past  Approaches,  Current  Trends,  and  Future 
Requirements,  in  Manned  Systems  Design,  edited  by  J.  Moraal  and  K.  F.  Krais, 
Plenum  Press,  New  York,  1981. 

**  Kline,  Melvin  B.,  Introduction  to  Systems  Engineering,  Lecture  Notes,  1982, 
Naval  Postgraduate  School,  Monterey,  Calif. 
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3.  Presents  the  procedures  used  in  estimating  the  "capability"  part  of 
effectiveness  that  have  been  derived  from  several  specific  studies 

5  X 

conducted  by  the  author.  Presentation  of  this  procedure  is  the 
major  purpose  of  the  report. 

The  specific,  analytic  efforts  referred  to  in  (3)  above,  were  conducted  in 
support  of  development  programs  and  make  provision  for  including  human 
operator  performance  in  systems  effectiveness  analysis.  When  the  methodology 
is  appropriate,  it  fulfills  the  need  pointed  out  in  references  2  and  3  above. 

It  should  be  added  that  there  is  little  "new"  material  in  this  report;  rather,  it 
collects  information  from  many  sources  and  focuses  it  on  how  to  conduct 
analysis. 

The  methodology  can  help  to  bridge  the  gap  between  the  human  factors 
specialist  and  the  systems  engineer.  It  can  be  used  to  quantify  the  importance  of 
the  operator's  performance  to  overall  system's  effectiveness,  and  thereby 
demonstrate  the  "worth"  of  including  human  factors  as  a  specialty  in  systems 
design. 


LIMITATIONS 

This  report  does  not; 

1.  Present  a  comprehensive  review  of  the  literature  in  systems 
analysis,  or  critique  other  analytic  methods  or  procedures. 

2.  Deal  with  cost-effectiveness.  Economic  factors  must  be 
considered  at  all  stages  of  system  design  and  acquisition. 
Cost-effectiveness  tradeoffs  must  be  performed,  but  we  are 
concerned  in  this  report  only  with  the  system  performance  part  of 
the  equation. 


Naval  Weapons  Center.  Launch  Opportunity  for  Air-to-Ground  Visually 
Delivered  Weapons,  by  Ronald  A.  Erickson  and  Carol  3.  Burge,  China  Lake, 
Calif.,  NWC,  January  1978.  (NWC  TP  6005,  publication  UNCLASSIFIED.) 
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Anti-Ship  Missile  Study;  Man-in-the-Loop  Operation,  by  Ronald  A. 
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3.  Explicitly  address  human  error  in  systems  operation.  No  methods 
are  presented  which  show  how  to  estimate  the  probability  of 
pushing  the  wrong  button,  or  making  the  wrong  turn.  Although 
very  important  in  system  design,  human  reliability  is  beyond  the 
scope  of  this  report. 

4.  Consider  the  human  factors  aspects  of  system  maintenance  and 
repair. 

The  scope  of  this  report  is  therefore  limited  further  in  relation  to  the  above 
USAF  definition  of  system  effectiveness.  We  are  concerned  with  predicting  a 
manned  system's  capability  (or  performance),  given  that  it  is  available  and 
dependable  (i.e.,  that  nothing  breaks).  Availability  and  dependability  will  not  be 
addressed. 


BACKGROUND 


The  following  section  is  intended  to  be  tutorial  in  an  overview  sense;  it  is 
intended  to  give  an  idea  of  what  a  system  is,  and  to  describe  what  is  done  in  the 
systems  analysis  process.  Selected  quotes  from  the  literature  are  used  when 
possible.  Other  quotes  on  the  same  subject  that  were  assembled  in  the  course  of 
this  study  are  given  in  Appendix  A.  The  purpose  of  including  Appendix  A  is  to 
give  the  inquisitive  reader  a  variety  of  comments  by  several  senior  specialists  in 
the  field. 


SYSTEMS 

This  study  deals  with  systems  which  include  human  operators.  Many 
definitions  of  such  systems  have  been  given  in  previous  publications;  some  are 
given  in  Appendix  A.  The  author's  definition  that  fits  the  topic  of  this  report  is 
given  below. 

A  man-machine  system  is  a  set  of  interacting  components 
composed  of  humans  and  machines  (including  software) 
directed  toward  performing  a  function  or  number  of  functions 
and  operating  within  the  constraints  of  time  and  specified 
environments. 

It  is  not  important  at  this  point  to  determine  if  the  above  definition  is 
"better"  than  the  others.  Suffice  it  to  say  that  the  specific  make-up  of  a  system 
is  in  the  eyes  of  the  beholder.  That  make-up  is  composed  of  physical  entities 
that  can  include  humans.  The  systems  we  are  concerned  with  here  are  designed 
to  perform  a  function  (or  functions),  and  we  are  concerned  with  whether  or  not, 
or  how  well  they  can  perform  that  function. 
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We  would  like  to  estimate  the  effectiveness  of  the  system  as  a  function  of 
the  operators'  abilities,  the  operating  conditions,  and  the  kind  of  tasks  and  tools 
(e.g.,  controls  and  displays)  in  the  system. 

In  statistical  terms,  we  would  like  to  estimate  the  variance  contributed  to 
system  effectiveness  by  the  operators  so  that  it  can  be  compared  to  the 
variances  contributed  both  by  major  components  in  the  total  system  and  by 
environmental  factors. 


SOURCES  OF  INFORMATION 

There  are  many  analytic,  or  mathematical  techniques  that  have  been  used  in 
the  prediction  of  system  performance,  and  specialties  have  emerged  around 
these  techniques:  systems  engineering,  systems  analysis,  and  operations 

research.  There  is  a  certain  amount  of  overlap  in  these  specialties,  as  illustrated 

in  Figure  1  (as  suggested  by  Singleton9).  Much  of  the  literature  can  be  grouped 
into  four  categories,  according  to  the  content  of  each.  Brief  comments  about 
these  four  categories  are  given  below. 


FIGURE  1.  Overlap  in  Specialties. 


Singleton,  W.  T.,  "Ergonomics  in  Systems  Design,"  ERGONOMICS  1967,  Vol. 
10,  No.  5,  pp.  541-548. 


P4 
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System  Development  and  Analysis*®’*  *’*^ 


These  types  of  books  outline  the  processes  to  be  followed  in  developing  a 
system  and  observations  are  made  about  each  step  in  the  process.  Man-machine 
factors  or  interfaces  are  sometimes  mentioned  explicitly,  and  some  analytic 
techniques  found  in  human  factors  (e.g.,  task  analysis)  are  mentioned,  but 
specifics  are  rarely  given.  These  texts  are  useful  references  when  working  as 
part  of  a  design  team. 


Human  Factors*^’*^’*^’*6’*7 

Most  of  the  articles  in  this  category  say  that  human  factors  should  be 
included  in  systems  development  procedures.  The  history  of  human  factors  and 
general  requirements  for  its  use  are  given.  Several  of  these  articles  are  of  t h 
"human  factors  people  should  .  .  .,"  or  "engineers  should  ...  "  variety.  T 

overview  articles  were  not  intended  to  relate  one-on-one  with  handbooks  t 
give  specific  human  performance  data.  Hopefully  this  report  will  provide  on 
the  missing  links  by  giving  all  of  the  analysis  procedures  to  follow  from  star  to 
finish. 


Chase,  Wilton  P.,  Management  of  System  Engineering,  John  Wiley  and  Sons, 
New  York,  1974. 

**  Wilson,  Warren  E.,  Concepts  of  Engineering  System  Design,  McGraw-Hill 
Book  Company,  New  York,  1965. 

12 

Chestnut,  Harold,  Systems  Engineering  Methods,  John  Wiley  and  Sons,  Inc, 
New  York,  1967. 

*■*  Meister,  David,  Behavioral  Foundations  of  System  Development.  John  Wiley 
and  Sons,  New  York,  1976. 

14 

Meister,  David,  and  Gerald  F.  Rabideau,  Human  Factors  Evaluation  in  System 
Development,  John  Wiley  and  Sons,  New  York,  1965. 

*^  Meister,  David,  "Systems  Development:  The  Future  of  Ergonomics  as  a 
System  Discipline,"  ERGONOMICS  1973,  Vol.  16,  No.  3,  pp.  267-280. 

*^  Jones,  J.  C.,  "The  Designing  of  Man-Machine  Systems,"  ERGONOMICS  1967, 
Vol.  10,  No.  2,  pp.  101-111. 

*^  DeGreene,  Kenyon  B.,  "Major  Conceptual  Problems  in  the  Systems  Manage¬ 
ment  of  Human  Factors/Ergonomics  Research,"  ERGONOMICS  1980,  Vol.  23,  No. 
1,  pp.  3-11. 
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Many  texts,  articles,  and  reports  discuss  the  analytic  or  mathematical  tools 
used  in  particular  applications  (e.g.  cueing  theory,  linear  programming).  These 
are  sometimes,  but  not  often,  put  in  an  overall  systems  development  context. 
Human  factors  analysis,  or  a  requirement  for  human  performance  data  is  seldom 
mentioned.  The  texts  are  necessary,  however,  in  determining  which  mathe¬ 
matical  techniques  to  use  in  an  analysis. 


A  Specific  Problem 


Technical  reports  document  studies  designed  to  solve  specific  problems  and 
sometimes  contain  useful  observations,  results,  or  generalizations  beyond  the 
immediate  problem.  These  observations  provide  background  material  and  may 
help  in  developing  guidelines  for  conducting  system/human  operator  analyses. 
Such  reports  also  can  serve  as  good  examples  of  how  to  do  a  specific  study,  if  the 
conditions  and  requirements  are  similar. 


SYSTEM  DEVELOPMENT  PROCEDURES 

A  specific,  "how-to"  document  that  is  the  objective  of  this  project  has  not 
yet  been  discovered.  Excerpts  from  selected  documents  that  might  prove  useful 
are  given  below,  without  the  detailed  definitions  of  terms  found  in  some  of  the 
references.  Hopefully,  the  gist  of  the  statements  and  diagrams  will  suffice  for 
the  moment. 


The  Concec 


20 

According  to  the  Human  Engineering  Guide  to  Equipment  Design,  the 
system  concept  of  design  and  development  is  the  concept  of  a  group  of 
components  designed  to  serve  a  given  set  of  purposes.  In  the  U.S.  Armed 
Forces,  a  set  of  purposes  is  called  a  mission;  in  industry,  the  purpose  might  be 
the  production  of  a  commodity  or  the  construction  of  a  facility.  In  any  case, 
system  design  is  the  design  of  a  total  system  so  that  it  serves  its  intended 
purposes  or  missions. 


Sivazlian,  B.  D.  and  L.  E.  Stanfel,  Analysis  of  Systems  in  Operations 
Research,  Prentice- Hall,  Inc.,  Englewood  Cliffs,  N.  3.,  1975. 

19 

Au,  Tung,  R.  M.  Shane,  and  L.  A.  Hoel,  Fundamentals  of  Systems  Engineering; 
Probabilistic  Models,  Addison-Wesley  Publishing  Company,  Reading,  Mass.,  1972. 

20 

Human  Engineering  Guide  to  Equipment  Design,  First  Edition,  Morgan,  Cook, 
Chapanis,  and  Lund,  Editors,  1963,  McGraw-Hill  Book  Company  Inc.,  New  York. 


System  design  includes  the  traditional  engineering  of  individual  equipments. 
A  set  of  procedures  must  be  followed  for  conceiving  and  developing  a  system  as 
a  whole  in  such  a  way  that  each  component  is  fashioned  to  make  its  proper 
contribution  to  the  ensemble.  Though  it  inevitably  involves  a  certain  amount  of 
traditional  "cut  and  try,"  it  can  be  a  rational,  orderly  process  of  analyzing  a 
system,  more  or  less  quantitatively,  before  it  exists,  then  designing  it,  and,  later, 
evaluating  the  system  in  its  prototype  or  preproduction  form. 

System  Analysis^ 

System  analysis  gives  as  accurate  a  picture  as  possible  of  the  structure  and 
functions  of  a  system  -  of  the  way  it  is  to  be  put  together  and  of  the  processes 
that  are  to  go  on  in  it.  System  analysis  is  a  part  of  the  system  design  process.  A 
system  analysis,  when  completed,  is  a  detailed  description  of  the  components  of 
a  system  and  of  the  operating  characteristics  of  those  components,  whether  men 
or  machines.  It  is  a  statement  of  the  capabilities,  limitations,  and  inter¬ 
dependencies  of  the  components  expressed  in  terms  that  are  relevant  to  the 
overall  mission  of  the  system. 


Procedures 
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Meister  and  Rabideau  list  the  procedures  in  doing  a  functional  analysis. 
Both  system  development  and  system  evaluation  depend  on  the  functional 
analysis  performed  in  the  predesign  and  early  design  phases.  Functional  analysis 
refers  to  the  processes  by  which  the  embryo  man-machine  system  is  planned. 
Functional  analysis  attempts  to  allocate  functions  between  men  and  machines,  to 
describe  the  tasks  performed  by  personnel  and  equipment,  and  to  specify  the 
criteria  to  be  utilized  in  system  design. 

Functional  analysis  is  accomplished  by  performing  the  following  steps: 

1.  Determining  the  system  mission  requirements. 

2.  Profiling  the  system  mission. 

3.  Segmenting  the  mission. 

4.  Identifying  and  describing  system  functions. 

5.  Establishing  functional  performance  criteria. 

6.  Allocating  functions. 

7.  Performing  a  task  analysis. 
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Functional  analysis  describes  the  system  mission,  resulting  in  a 
determination  of  system  functions  and  functional  criteria.  The  human  factors 
analysis  is  usually  performed  within  the  framework  and  as  a  part  of  an  overall 
system  analysis,  with  the  human  engineer  participating  as  one  member  of  a  team 
composed  primarily  of  engineers. 


Procedure  Diagrams 

Procedures  to  be  followed  in  the  system  analysis  process  have  been 
diagrammed  by  many  authors;  several  examples  are  shown  in  Appendix  A  to 
illustrate  how  different  people  view  the  process.  Some  are  useful,  some  aren't. 
A  simplified  diagram  that  will  serve  as  a  starting  point  in  this  report  is  shown  in 

Figure  2.  More  detailed  steps  in  the  process  will  be  given  later  in  this  report 
as  the  analysis  process  is  developed. 

In  summary,  this  background  section  has  intended  to  give  the  reader,  (1)  an 
idea  of  what  a  system  is,  and  (2)  the  analytic  processes  that  have  been  used  in 
the  development  of  a  system. 


Air  Force  Systems  Command,  Measures  of  Effectiveness  Literature  Survey, 
by  Robert  A.  Hermann,  April  1974,  DCS/Development  Plans,  AFSC,  Kirtland 
AFB,  NM,  OAS-WP-74-2. 
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SYSTEM  PERFORMANCE 

This  section  of  the  report  will  discuss  some  concepts  in  systems  perfor¬ 
mance,  and  the  operator's  contribution  to  it. 


WHY  PREDICT  PERFORMANCE? 

The  best  way  to  find  out  how  well  a  system  works  is  to  measure  its 
performance;  this  is  usually  done  in  a  test  facility  (or  on  a  range)  where  the 
necessary  measuring  equipment  is  available.  The  effectiveness  of  existing 
systems  must  be  predicted,  however,  for  conditions  where  there  are  no  test  data, 
where  testing  is  not  a  feasible  way  to  indicate  performance,  or  where  an  existing 
system  (the  "baseline")  is  compared  to  a  new,  hypothetical  system  (the 
"improvement"). 

The  effectiveness  of  hypothetical,  or  proposed  systems  is  predicted  to 
provide  data  for  decision-making  at  many  levels:  from  specification  of  system 
components  (e.g.,  the  field-of-view  of  an  optical  sensor),  all  the  way  up  to 
deciding  whether  a  new  system  should  be  procured  at  all.  The  characteristics  of 
these  hypothetical  systems  must  be  specified  in  enough  detail  to  allow  them  to 
be  modeled.  The  human  operator's  role  in  system  operation  must  also  be 
specified. 


MEASURE  OF  EFFECTIVENESS  (MOE) 


The  next  few  sections  of  this  report  will  discuss  the  items  in  the  boxes  of 
Figure  2.  After  that  will  follow  the  specifics  on  how  to  implement  Figure  2. 

The  products  of  a  system  effectiveness  analysis  are  usually  numbers  of  some 
sort  that  describe  how  well  the  system  performs  its  functions.  It  is  important 
that  these  numbers,  or  measures  of  effectiveness  (Box  3,  Figure  2)  be  useful  to 
the  sponsor,  user,  or  decision  maker  for  whom  the  analysis  was  done.  A  textbook 
or  primer  on  developing  MOEs  was  not  found  in  the  course  of  this  study,  and  yet 
selecting  the  proper  MOEs  is  very  important  in  any  analysis. 


Example 


Let  us  assume  that  a  new  transport  system  is  being  proposed;  it  would 
transport  individuals  via  surface  vehicle  from  place  to  place  within  a  city.  Some 
MOEs  for  this  system  are  shown  in  Table  1. 


Are  the  example  MOEs  in  Table  1  useful  to  the  decision  makers  and/or 
design  engineers?  Are  they  quantifiable?  How  can  they  be  modeled  or 
estimated?  How  many  MOEs  should  be  used? 
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TABLE  1.  Some  MOEs  for  a  Transport  System. 

1.  Percent  of  people  desiring  transportation  at  any  one  time  that  can 

be  transported. 

2.  Timeliness  in  maintaining  schedule. 

3.  Amount  of  time  spent  by  passengers  waiting  for  next  bus. 

4.  Speed  of  transportation. 

5.  Accessibility  of  bus  stops  to  homes,  offices,  etc. 

6.  Accident  rate  on  transportation  system. 

7.  Crime  rate  on  transportation  system. 

8.  Comfort  during  transportation. 

Good  characteristics  for  an  MOE  found  in  Military  Operations  Research 
Society  documentation  include: 

a.  Relevancy  to  the  mission 

b.  Relevancy  to  the  design  issues 

c.  Relevancy  to  the  decision-makers 

d.  Quantifiable 

The  analyst  working  on  a  problem  at  any  level  must  realize  his  MOE  may  be 
used  by  another  analyst  working  at  the  next  higher  level.  Each  analyst  must, 
therefore,  understand  fully  how  his  analysis  will  be  used  at  levels  above  and 
below  him. 

There  are  different  MOEs  for  the  various  aspects  of  system  employment  and 
these  different  MOEs  must  all  be  relevant  to  the  success  of  system  employment. 
The  human  factors  analyst  is  rarely  given  the  detailed  MOEs  with  which  he  must 
be  concerned;  he  must  identify  them  himself.  These  MOEs  should: 

1.  Be  "approved"  through  a  process  of  iteration  and  negotiation  with  the 
program  managers  and  other  analysts. 

2.  Be  required  in  some  decision  process  (e.  g.,  component  selection). 
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3.  Be  clearly  stated  in  the  final  study  report,  with  the  rationale  for 
selection  also  given. 

4.  Include  aspects  of  the  physical  environment  that  affect  operator  and 
system  performance. 

5.  Use  variables  that  are  readily  measurable  in  the  real  world,  and/or  for 
which  there  is  a  data  base. 

6.  Be  useable  by  other  analysts  in  the  program. 


Guidelines  For  Developing  MOEs 

MOEs  are  created  for  specific  circumstances  and  must  be  backed  up  by 

experienced  judgement.  A  suggested  procedure  for  the  development  and 

22 

application  of  MOEs  is  given  by  Anderson,  et  al,  ; 

a.  Develop  or  create  as  many  sensible  MOEs  as  possible. 

b.  Categorize  them  into  groups  of  similar  measures. 

c.  Select  the  best  MOEs  in  each  group  using  a  procedure  to  evaluate 
those  that  are  strong  or  weak,  or  alike  or  similar. 

d.  Point  the  selected  MOEs  to  the  next  higher  level  of  objectives: 
i.e.,  insure  that  the  MOEs  are  so  constructed  that  they  can  serve  as 
performance  indicators  to  the  next  higher  level. 

e.  Express  the  MOEs  in  standard  notation  of  physics,  engineering,  and 
mathematics  (i.e., time  required,  distance  covered,  percent  defects 
identified). 

A  similar  process  is  to  specifically  relate  MOEs  to  mission  objectives. 

a.  List  important  mission  features,  so  that  the  MOE  will  have  a  better 
chance  of  reflecting  the  way  a  mission  must  be  conducted  in  order 
to  be  effective. 

b.  Develop  an  extensive  list  of  conceivable  MOEs  for  the  mission  (no 
initial  constraints  should  be  put  on  this  brainstorming). 


U.S.  Army  Combat  Developments  Command,  Force  Developments;  The 
Measurement  of  Effectiveness,  by  D.  C.  Anderson,  et  al,  January  1973,  Ft. 
Belvoir,  VA,  USACDC  Pamphlet  No.  71-1. 
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c.  Reduce  the  list  by  discarding  duplication  and  MOEs  that  are  not  in 
some  way  related  to  the  mission  objective. 

d.  Write  a  brief  discussion  of  each  of  the  MOEs  and  give  the  analyst's 
views  of  some  of  the  general  characteristics  of  each  MOE. 

An  example  of  the  product  of  the  above  procedure  is  shown  in  Table  2.  The 
headings  across  the  top  of  Table  2  could  be  used  in  any  MOE  development. 


TABLE  2.  General  Characteristics  of  Candidate  MOEs 
for  Close  Air  Support  (CAS)  Mission. 


Candidate  MOE 


Relation  to 
mission 
objective 

Systems  that 
can  be 
compared 

Typical  availability 
of  data 

Inherent  assumptions 

Ways  the  measure 
can  be 
misleading 

Direct 

All 

"Casualty-producing" 
data  are  required 
for  enemy  forces 
and  enemy  tactics. 

In  some  situa¬ 
tions.  time  to 
achieve  objective 
may  be  impor¬ 
tant  to  coordinate 
multiple  ground 
force  operations. 

Implied 

Those  that 
are  similar 
in  two  of 
the  three 
parameters. 

fair,  predictability 
of  response  time 
is  usually  suspect. 

Enemy  lossed  produce 
a  corresponding  re¬ 
duction  in  friendly 
casualties.* 

Weapons  will  not 
endanger  friendly 
forces. 

Ration  allow  one 
parameter  to 
overpower  others. 

Implied 

All 

Poor:  response  time 
often  depends  on 
people  and  varies 
greatly. 

Systems  that  provide 
rapid  response  will 
be  able  to  achieve 
mission  objectives.* 
Individual  situations 
are  short-term  and 
duration  of  support 
is  not  too  impor¬ 
tant. 

Long-term  benefits 
of  slower  but 
more  lethal  weap¬ 
ons  may  be  over¬ 
looked. 

Implied 

All 

Enemy  is  resourceful 
and  will  take  advan¬ 
tage  of  times  we 
cannot  supply  CAS  * 

A  low-performance 
weapon  with  all- 
weather  capability 
may  not  reduce 
our  losses. 

Implied 

All 

Very  poor,  there  are 
no  known  data  to 
support  time 
estimates:  also, 
situations  vary 
greatly. 

Casualties  are  directly 
related  to  time 
required  to  handle 
a  particular 
situation.* 

Losses  inclined 
while  achieving 
objectives  are  not 
necessarily 
examined. 

Reduction  to  over¬ 
all  losses  to 
friendly  forces 
while  they  are 
being  assisted. 


(Our  fire  support 
X  duration  of 
supporl)/re- 
sponse  time. 


Response  time 


T-  of  time  the 
system  can  be 
used. 


Time  for  ground 
forces  to  achieve 
their  objectives. 
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If  more  than  one  MOE  is  used,  as  is  usually  the  case,  a  scheme  must  be 
devised  for  establishing  relative  priorities.  Several  MOEs  can  be  combined  with 
weighting  factors  to  produce  a  single  "summary"  MOE. 

In  summary,  the  determination  of  the  measures  of  effectiveness  is  an 
important  early  step  in  the  analysis  process.  There  are  no  set  rules,  but  the 
guidelines  given  above  are  useful.  The  procedures  in  determining  the  MOE  will 
be  discussed  in  more  detail  later  in  this  report. 


FACTORS  AFFECTING  SYSTEM  PERFORMANCE 

The  factors  affecting  system  performance  (Box  4,  Figure  2)  can  be  broken 
down  in  several  ways;  one  such  way  is  shown  in  Table  3.  The  operator's 
performance  requirements  are  determined  by  the  man-machine  function 
allocation  and  will  be  discussed  in  more  detail  in  the  next  section. 


TABLE  3.  Breakdown  of  Factors  Affecting  Overall  System  Performance, 
with  Examples  of  Subdivisions. 


1.  Operator  Performance/Capabilities 

a.  Search  Time 

b.  Tracking  Accuracy 

c.  Data  Entry  Time 

2.  System  Characteristics 

a.  Software 

b.  Hardware 

c.  Capabilities 

3.  Operator's  Environment/Operating  Conditions 

a.  Workload 

b.  Time  Available 

c.  Physical  Environment 

4.  System's  Environment/Operating  Conditions 

a.  Weather  (e.g.  visibility) 

b.  Communication  Interference  (e.g.  jamming) 

c.  Obstacles/Threats 


etc 
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The  system  characteristics  in  Table  3  describe  the  system  such  that  its 
capabilities  can  be  predicted.  The  analyst  only  needs  to  go  to  the  level  of  detail 
required  to  predict  capabilities.  As  an  example,  "The  engine  is  such  that  a 
velocity  of  80  mph  can  be  sustained  for  3  hrs"  is  sufficient.  Whether  it  is  a 
gasoline  or  diesel  engine  would  not  be  relevant  to  operator/system  performance 
effectiveness  estimates. 

A  description  of  the  operator's  environment  is  necessary  to  estimate  his/her 
performance,  and  will  be  discussed  in  the  next  section.  As  an  example,  the  time 
available  to  do  the  job  might  be  related  to  vehicle  velocity,  orbit  period,  or 
conveyor  belt  speed.  Aircraft  maneuvering  requirements  determine  the  "g"  level 
under  which  tasks  must  be  performed.  Vehicle  vibration  would  affect  manual 
tracking  accuracy. 

The  system's  environment  (Item  4  in  Table  3)  can  be  different  from  that  of 
the  operator,  and  also  affects  system  performance.  Many  systems  are  expected 
to  function  under  a  wide  range  of  conditions;  for  example,  temperatures  from 
-50°F  to  +130°F,  humidities  from  0  to  100%,  and  altitudes  from  0  to  60,000  ft. 
Those  conditions  that  affect  the  performance  of  the  system  must  be  included  in 
the  analysis.  For  example,  the  roughness  of  the  terrain  directly  affects  how  fast 
a  vehicle  can  move  across  it.  This  factor  must  be  included  in  personnel  carrier 
design  studies. 


Frequency  of  Occurrence 

The  distribution  of  the  values  of  a  factor  that  affects  performance  is  used 
as  a  weighting  term  in  the  overall  effectiveness  calculation.  These  distributions 
are  difficult  to  obtain  in  many  cases;  estimates  rather  than  actual  data  must 
often  be  used. 

As  an  example,  assume  that  a  system  must  operate  in  various  types  of 
terrains  and  weathers,  and  it  has  been  established  that  both  terrain  type  and 
visibility  affect  performance.  Tables  4  and  5  show  hypothetical  distributions  of 
anticipated  employment  environments.  If  terrain  type  and  visibility  are 
correlated  (e.g.,  the  desert  is  mostly  clear),  this  "interaction"  must  also  be  taken 
into  account  in  the  weighting  procedure. 

The  analyst  must  tabulate  the  factors  (or  variables)  affecting  system 
performance,  specify  the  range  of  the  variables  (e.g.,  0  to  60,000  ft),  and  most 
important,  determine  the  distributions  of  values  of  the  variables. 

These  variables  can  then  be  used  in  calculating  performance;  a  common 
result  would  be  the  percent  of  the  time  that  the  system  could ’be  used,  or  the 
parts  of  the  world,  or  percent  of  terrains  where  its  employment  would  be 
satisfactory.  These  items  are  essentially  the  MOEs  that  have  been  determined 
earlier  in  the  analysis. 
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PREDICTING/ESTIMATING  SYSTEM  COMPONENT  PERFORMANCE 

The  following  two  sections  deal  with  how  to  get  data  about  the  system  and 
its  environment  (Box  5,  Figure  2). 

Figure  3  illustrates  four  concepts  of  how  component  performance  has  been 
quantified  by  different  analysts.  In  (A),  the  simplest,  a  given  input  (e.g.,  light) 
makes  it  possible  for  the  component  to  produce  an  output  (e.g.,  a  photograph). 
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In  concept  B,  interacting  factors  are  included  and  affect  the  output.  In  the 
operation  of  a  camera,  the  operator's  steadiness,  and  the  sun  angle  or  haze 
condition  would  affect  the  output.  In  concept  C,  the  internal  operation  of  the 
component  (e.g.,  automatic  light  meter)  is  considered  in  estimating  the  output. 
If  one  is  attempting  to  estimate  the  effect  of  the  internal  workings  of  the 
component  upon  system  performance,  then  concept  C  is  useful. 

In  concept  D,  the  interacting  factors  are  added.  Concept  D  would  produce 
the  best  estimate  of  performance,  if  it  can  be  implemented  properly.  Concept  B 
includes  the  system  aspects  of  performance  and  would  be  the  second  choice. 

Several  methods  of  getting  estimates  of  component  performance  can  be 
used:  mathematical  models,  laboratory  measurements,  simulations,  and 
measurements  made  in  the  real  world.  These  methods  overlap,  and  sometimes 
compliment  one  another.  There  are  no  standard  definitions  of  the  terms,  but  the 
point  to  be  made  is  that  there  is  a  continuum  of  estimation  methods. 

An  example  of  this  continuum  and  its  application  to  Figure  3  is  shown  in 
Table  6,  which  contains  subjective  estimates  from  two  people  with  experience  in 
target  acquisition  systems.  The  entries  could  well  change  for  other  functions 
performed  by  other  types  of  systems. 

TABLE  6.  An  Example  of  the  Usefulness  of  Methods 
of  Estimating  Performance. 


Method  of  Component  Performance  Concept 

Predicting  - 

Performance  A  B  C  D 


Math  Models* 

Poor 

Medium 

Good 

Medium 

Laboratory  Measurement 

Good 

Medium 

Good 

Medium 

Man-in-the-Loop  Simulation 

Poor 

Good 

Poor 

Good 

Field  Trials 

Poor 

Good 

Poor 

Medium 

Operational  Performance 

Poor 

Medium 

Poor 

Poor 

*  or  computer  simulation  methods 

The  observations  that  can  be  made  about  Figure  3  and  Table  6  are: 

1.  Concepts  A  and  B  are  empirical.  We  don't  know  what's  inside  each  box, 
and  hence  can't  model  it.  We  don't  know  how  interacting  factors  will  change  the 
output  in  concept  B  because  we  can't  "model"  how  it  operates  on  its  inputs.  We 
can  only  measure  the  output. 


2.  The  use  of  Concepts  A  and  C  in  simulation,  field  testing,  or  operational 
data  collection  does  not  produce  the  data  one  usually  desires,  since  the  concepts 
do  not  include  "outside",  interacting  factors. 

In  summary,  estimating  component  performance  can  treat  the  component  as 
a  "black  box, "or  consider  its  internal  make-up.  If  interacting  factors  such  as 
the  operator  or  the  outside  environment  have  a  strong  influence  upon  the 
component's  performance,  they  should  be  included. 

There  are  many  ways  that  this  performance  estimation  can  be  made,  ranging 
from  math  models  to  field  tests.  The  selection  of  both  concept  and  method  is  a 
subjective  process  influenced  by  time,  cost,  and  accuracy  requirements,  but  it  is 
a  process  that  should  be  consciously  carried  out  and  documented.  Information 
like  that  contained  in  Figure  3  and  Table  6  can  help  the  analyst  chose  the  proper 
approach.  He  should  build  his  own  Table  6  with  inputs  from  experts  on  his 
particular  application. 


HUMAN  OPERATOR  PERFORMANCE 


The  operator's  performance  at  the  different  tasks  required  in  system 
operation  must  also  be  measured  or  estimated.  The  data  are  represented  in 
Boxes  5  and  6  of  Figure  2. 

The  concepts  of  measurement  of  the  performance  of  the  human  operator  are 
similar  to  those  shown  in  Figure  3  and  are  shown  in  Figure  4.  A  task  description, 
or  instructions  to  the  operator,  is  shown  as  the  input.  The  operator  performs  the 
task  using,  in  some  way,  the  system  component(s).  This  could  be  a  display 
(target  recognition),  a  control  stick  and  display  (tracking),  or  a  steering  wheel, 
clutch,  gear  shift,  brake,  etc.  (driving).  The  man/machine  combination  can  be 
considered  as  a  unit,  as  separate  units,  or  each  as  a  number  of  sub-components. 

Performance  can  be  estimated  with  or  without  consideration  of  environ¬ 
mental  factors.  In  addition,  the  operation  of  a  system  usually  involves  many 
tasks  with  different  system  components,  so  many  such  boxes  would  be  required 
to  represent  system  operation.  For  example,  aircraft  flight  involves  "steering" 
the  aircraft,  operating  the  flight  system,  the  radar  system,  and  the  navigation 
system,  as  well  as  communicating. 

There  are  other  important  factors  not  shown  in  Figure  4:  operator  training, 
skill  level,  and  motivation.  Levels  of  these  factors  should  be  specified  when 
appropriate  (and  possible).  They  are  considered  to  be  part  of  the  operator  in 
Figure  4  and  are  not  shown  separately. 
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PERFORMANCE 


FIGURE  4.  Operator  Performance  Concepts. 


OPERATOR 


Some  personnel  stereotypes  could  be  associated  with  the  boxes  in  Figure  4, 
although  as  with  all  stereotypes,  the  association  is  not  accurate  all  the  time,  and 
is  sometimes  unjust.  The  "academic"  analyst  sometimes  uses  Figure  4A.  The 
experienced,  "real-world"  analyst  uses  4B,  since  he  is  aware  that  environmental 
factors  can  be  very  important.  Some  engineers  prefer  4C;  they  like  to  tinker 
with  the  hardware.  Psychologists  like  4D  where  human  modeling  is  stressed,  and 
mathematical  modelers  and  programmers  prefer  4E  or  H  since  everything  in  the 
world  can  be  modeled! 


DESCRIBING  THE  TASKS 

Before  operator  performance  data  can  be  found  or  generated,  the  tasks, 
system  components,  and  important  environmental  factors  must  be  described 
(Boxes  2  and  4,  Figure  2).  A  number  of  human  factors  techniques  that  can  be 

13  23  24 

used  are  well  documented  by  Meister,  *  by  Coburn,  and  in  The  Human 

25 

Engineering  Guide  to  Equipment  Design. 

The  task  description  used  in  system  effectiveness  studies  for  simple  tasks 
need  not  be  as  detailed  as  is  required  for  specific  equipment  design  studies.  For 
example,  the  shape,  size,  or  kind  of  control  (button  or  switch)  need  not  be 
specified.  It  can  be  assumed  that  the  most  appropriate  control  will  be  selected 
by  later  human  factors  design  studies.  An  estimate  of  the  distribution  of  times 
required  to  operate  the  control  may  be  the  only  description  required,  for 
instance. 

Complex  tasks,  which  are  often  affected  more  by  outside  influences  (e.g., 
the  weather)  need  to  be  described  in  more  detail  to  make  sure  that  the 
performance  data  used  in  the  analysis  is  appropriate. 


OUTSIDE  FACTORS  AFFECTING  TASK  PERFORMANCE 

Both  Figures  3  and  4  show  interacting  or  environmental  factors  in  some 
concepts  as  inputs  which  affect  performance.  These  outside  factors  need  not  be 
considered  part  of  the  system,  and  are  usually  not.  But  they  must  be  included 
when  it  comes  to  estimating  performance. 


Meister,  David,  Human  Factors;  Theory  and  Practice.  John  Wiley  and  Sons, 
Inc.,  New  York,  1971. 

24 

Naval  Electronics  Laboratory  Center,  Human  Engineering  Guide  to  Ship 
System  Development,  by  R.  Coburn,  3  October  1973,  NELC/TD  278,  San  Diego, 
CA. 

25 

Human  Engineering  Guide  to  Equipment  Design,  edited  by  H.  P.  Van  Cott  and 
R.  G.  Kinkade,  Superintendent  of  Documents,  2nd  Edition,  U.  S.  Government, 
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Factors  affecting  performance  are  represented  by  Box  4  in  Figure  2.  They 
can  be  divided  into  three  categories,  with  the  inevitable  overlap  one  finds  with 
any  such  taxonomy.  System  characteristics  are  those  things  that  the  engineer 
can  solder,  measure,  program,  polish,  or  pound  on.  Operating  conditions  indicate 
how  the  system  or  subsystem  will  be  used,  and  environmental  factors  describe 
where  or  when  it  will  be  used.  Examples  of  these  categories  are  given  in  Table 
7.  A  list  similar  to  Table  7  should  be  made  up  in  the  initial  stages  of  the 
analysis. 

Interrelationships  or  independence  of  these  factors  must  also  be  considered. 
Some  factors  will  affect  the  equipment  operation  independent  of  the  operator 
(e.g.,  haze  will  reduce  the  quality  of  a  photograph),  and  others  primarily  will 
affect  the  operator's  ability  to  perform  (noise  or  ambient  temperature  in  the 
workspace).  There  are  also  factors  which  can  affect  both  equipment  and 
operator  (vibration). 

It  is  important  to  include  these  factors  in  the  task  description  so  that  the 
search  for  or  generation  of  data  will  produce  a  good  estimate  of  performance. 
The  human  factors  literature  and  handbooks  (reference  29)  list  most  such  factors 
and  give  some  idea  of  how  they  have  affected  performance  in  the  past.  A  few 
examples  are  given  in  Table  8. 


OPERATOR  CHARACTERISTICS 

Figure  4D  and  E  show  the  operator  as  something  that  could  be  described  or 
modeled.  Many  operator  models  have  been  developed  and  can  be  used  when 
appropriate  (e.g.  manual  tracking  or  visual  performance).  Factors  such  as  skill 
level,  pertinent  experience,  fatigue,  training,  and  motivation  are  important;  if 
they  can  be  defined  and  quantified,  they  should  be  included  in  the  analysis.  At 
the  very  least,  assumptions  as  to  these  factors  should  be  stated  in  the  analysis 
documentation. 


System  Characteristics 

Audio  Tone  Characteristics  (Frequency,  Volume) 
Display  Size  and  Viewing  Distance 
Display  Format  (Symbols,  Color) 

Field-of-View  of  Optics  or  Windows 
Sensor  Characteristics  (Resolution) 

Steering  Control  Size  and  Configuration 
Slew  Controller  Size  and  Shape 
Tracking  Loop  Design  (Rate- Aided) 

Workspace  Layout 
Operating  Conditions/Requirements 


Vehicle  or  Aircraft  Velocity 
Altitude  of  Flight  or  Depth  Under  Water 
Accelerations  to  be  Encountered 
Time  Available  to  Perform  Task(s) 

Task  or  Mission  Duration 
Distance  to  be  Traversed 
Environmental  Factors 

Wind  Velocity 
Air  Turbulence 
Temperature,  Humidity 
Outdoor  Light  Level  (Day,  Night) 
Visibility  or  Atmospheric  Transmission 
Terrain  Roughness 
Water  Roughness  (Sea  State) 

Slickness  of  Driving  Surface 
Traffic  Density 
Electronic  Jamming 
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TABLE  8.  Examples  of  Operator  Tasks  and  Outside  Factors 
That  Can  Affect  Performance. 


Task  Outside  Factors 


Visual  Perception 

Visibility 

Detecting 

Occluding  Objects 

Reading 

Vibration 

Identifying 

Acceleration 

Searching 

Camouflage 

Manual  Tracking 

Vibration 

Acceleration 

Visibility 

Aural/Oral  Communication 

Vibration 

Acceleration 
Background  Noise 
Jamming 

Decision-Making 

Number  of  alternatives 
Value  or  utility  of  each 
alternative 

All  of  the  Above 

Training/Experience 
Expectation 
Consequence  of  Error 
Motivation 

Time  Available 
Workload 

Mission  Duration 

THE  FORM  OF  PERFORMANCE  DATA 


THE  RAW  DATA 

Human  operator  performance  data  used  in  analysis  has  taken  many  forms: 
average,  mean,  or  median  scores,  frequency  distributions,  and  cumulative  plots. 
The  raw  data  generated  by  field  tests,  simulations,  or  laboratory  experiments 
can  take  the  form  shown  in  Figure  5.  This  data  form  implies  that: 

1.  a  given  test  condition  has  been  repeated  a  number  of  times, 

2.  a  number  of  different  operators  have  been  used  in  the  tests, 

3.  or,  the  same  operator  was  used  in  a  number  of  test  trials. 
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It  should  be  stressed  here  that  the  data  shown  in  Figure  5  is  typical.  There 
is  always  a  distribution  in  scores,  not  just  one  value.  This  distribution,  or 
variability,  is  caused  by  differences  between  people,  or  within  the  same  person 
at  different  times.  Other  causes  can  contribute  to  the  variability  that  we  cannot 
control,  identify,  or  measure.  Suffice  it  to  say  that  a  distribution  in 
performance  scores  is  a  natural  phenomenon,  and  we  must  take  it  into  account  in 
our  analysis  or  we  will  get  spurious  answers. 
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FIGURE  5.  Distribution  of  the  Range  at  Which  an  Aircraft 
is  Sighted  for  a  Specific  Environmental  Situation. 


Figure  6  shows  the  same  data  plotted  in  cumulative  form.  The  curve  in 
Figure  6  represents  performance  under  one  set  of  conditions.  A  large  number  of 
such  curves  would  be  produced  in  an  experiment  where  several  parameters  were 
varied. 


THE  TRADITIONAL  DATA 

Many  curves  similar  to  the  one  in  Figure  6  are  usually  summarized  by 
picking  one  point  off  of  each  curve  (the  dotted  line  in  Figure  6)  to  use  as  an 

indicator  of  performance.  The  rationale  for  this  is  given  by  Taylor:^ 

"It  is  found,  upon  plotting  many  hundreds  of  such  stimulus 
presentations,  that  the  probability  of  target  detection  rises 
with  stimulus  magnitude  in  accordance  with  an  ogive  curve 


Taylor,  John  H.  "Use  of  Visual  Performance  Data  in  Visibility  Prediction," 
Applied  Optics  1964,  Vol.  3,  No.  5,  p.  562. 
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which  is  well  fitted  by  a  normal  Gaussian  integral.  Statisti¬ 
cally,  the  best  determined  point  of  the  ogive  is  the  point  of 
inflection,  i.e.,  where  the  probability  of  correct  discrimination 
is  0.50,  and  this  is  the  value  of  threshold  contrast  of  prime 
interest  in  laboratory  studies." 

The  above  procedure  is  standard  in  well  controlled  laboratory  studies,  and 
analysts  often  use  such  averages,  means,  or  medians  computed  from  raw  data. 
The  analyst  should  not  use  such  averages  in  system  effectiveness  analysis, 
however;  the  result  can  be  very  misleading.  If  a  median  is  used  as  the 
performance  data,  50%  of  the  operators  may  do  better  than  required  by  the 
design;  but  50%  will  do  worse.  Hence,  the  system  may  be  optimized  for  only  the 
best  half  of  the  operators!  Data,  in  the  form  shown  in  Figure  6,  must  be  used. 
The  problem  is  to  find  such  relatively  complete  data  from  past  experiments  or 


RANGE,  THOUSANDS  OF  FEET 


FIGURE  6.  Normalized  Cumulative  Plot  of  the  Distribution  in 
Sighting  Ranges  Shown  in  Figure  15.  This  figure  is  usually 
called  the  cumulative  percentage  of  detection  range. 


OBTAINING  PERFORMANCE  ESTIMATES 


The  analyst  would  like  to  obtain  the  operator  performance  estimates 
without  having  to  generate  them  himself. 

Estimates  can  be  obtained  from  theoretical  studies,  computer  simulation 
models,  laboratory  experiments,  man-in-the-ioop  simulations,  or  field  tests. 
Figure  7  shows  such  a  continuum,  with  finer  divisions  in  the  procedures  than  are 
shown  in  Table  6. 


[i] 


NWC  TP  6541 


Sources  for  Truck  and  Aircraft  Design  Studies. 
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Although  the  terms  "estimates"  and  "data"  are  used  interchangeably  here, 
the  purist  might  argue  that  theory  and  models  produce  calculations  and 
experiments  and  field  test  produce  "real  data"! 

The  quality  or  applicability  of  the  estimates  varies  according  to  the  source 
and  the  system  or  tasks  in  question.  In  other  words,  each  different  technical 
area  will  have  its  own  source  of  information  for  use  in  estimating  performance; 
some  technical  areas  have  more  and  better  data  than  others. 

Table  9  shows  an  example  of  an  evaluation  of  data  quality  for  a  specific 
application.  Such  a  table  can  be  made  for  any  technical  area  by  those  working  in 
the  particular  field  and  used  as  a  guide  in  planning  analysis  and  validation  tests. 

TABLE  9.  Example  of  Operator  Tasks  and  Performance  Measures, 
and  "Best  Bet"  Data  Sources  for  a  Target  Acquisition  System. 


Operator  Tasks/ 
Performance 


Weapon  Delivery  Accuracy 

Target  Detection 

Target  Classification 

Target  Identification 

Range  at  Detection  (D) 
Classification  (C) 
Identification  (I) 

Time  to  Operate  System 

Time  to  D/C/I 

False  D/C/I 


Laboratory  Flight 

Analysis  Experiments  Simulations  Tests 


Data  must  be  selected  by  somehow  balancing  applicability  and  validity.  The 
best  source  is  tests  done  expressly  for  the  analysis  being  performed.  The  second 
source  would  be  data  from  tests  or  simulations  dealing  with  a  similar  system. 
Multiple  sources  can  also  be  used  (including  interviews  with  experienced 
operators);  if  a  consensus  emerges,  so  much  the  better.  Selection  of  data  is  a 
subjective  process  where  experience  both  in  analysis  and  in  conducting  human 
factors  experiments  helps.  It  is  important  that  the  data  sources  be  documented, 
so  that  the  results  can  be  explained  to  reviewers  of  the  analysis,  and  so  that 
additional  data  can  be  comprehensively  included  at  a  later  date. 
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DEVELOPING  A  SYSTEM  EFFECTIVENESS  ALGORITHM 


ITERATIONS  REQUIRED 

Two  types  of  iterations  are  required  throughout  this  analysis;  personnel 
iterations  and  technical  iterations.  The  participants  in  a  system  development 
effort  are  listed  in  Table  10;  discussions  among  some  of  these  participants  is  a 
requirement  at  various  stages  of  the  analysis  process.  Agreement  or  approval 
with  what  has  been  produced  is  the  goal  of  these  discussions.  The  discussions 
will  result  in  changing  some  of  the  analysis:  the  iteration  process. 

Some  of  the  iteration  points  will  be  identified  below.  Others  are  of  a 
non-technical  nature  and  will  not  be  included  in  this  report. 


OVERVIEW  DESCRIPTION 

The  procedures  in  conducting  the  analysis  have  already  been  described  in 
general  above.  The  first  step  is  to  document  the  mission  and  system  concepts 
that  would  be  found  in  Boxes  1  and  2  of  Figure  2.  This  Overview  Description  is 
based  upon  work  that  has  presumably  already  been  done  but  sometimes  is  not 
adequately  documented. 

Doing  that  initial  work  (i.e.,  determination  of  requirements;  system  concept 
formulation)  is  not  the  subject  of  this  report;  it  is  assumed  here  that  the  work 
has  been  done,  and  must  be  documented  in  a  form  suitable  to  the  analytic 
process. 

Table  11  shows  the  first  steps  in  describing  the  situation.  Items  in  the  table 
can  be  related  to  previous  sections  of  this  report,  and  have  already  been 
discussed  at  length. 

The  design  of  a  bus  for  public  transportation  will  be  used  as  an  example 
throughout  the  remainder  of  this  report.  An  example  such  as  a  fighter  or  attack 
aircraft  or  a  weapons  system  was  intentionally  not  used  in  this  study.  It  was  felt 
that  using  something  that  the  author  has  not  worked  with  before  would  produce  a 
better  generalization  of  the  techniques.  Table  12  shows  the  application  of  Table 
11  to  the  design  of  the  transport  system;  it  is  not  meant  to  be  complete  in  itself, 
but  only  to  bring  Table  1 1  into  the  real  world. 
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TABLE  10.  Example  of  the  Participants  in  the  System 
Effectiveness  Analysis  Process. 


m 

Participants 

Example  of  Specialty 

Remarks 

1.  Analysts 

Operations  Research 

Members  of  the  concept, 

tv  : 

Systems  Analysis 

design,  and  evaluation 

Human  Factors 

team. 

1 

2.  Engineers 

Mechanical 

Electronic 

Optical 

Principal  members  of 
the  design  team. 

Aeronautical 

Human  Factors 

»£V 

.>  .*• 
* ^  . 

.  v  / 

3.  Scientists 

Atmospheric 

Usually  consultants. 

/. 

a 

Oceanographic 

w 

Psychology 

Physics 

1  •_ 

Chemistry 

Mathematicians 

Computer  Scientists 

4.  Current  Users 

Pilot 

May  not  be  near 

Fireman 

design  team.  Also, 

Truck  Driver 

they  may  not  under- 

K*  v 
VV 

Farm  Worker 

stand  the  design 

i.*  ■: 

process. 

> 

5.  Managers 

Branch  Chief 

Coordinate  and  direct 

Project  Head 

work. 

Program  Manager 

i  *  -  - 

Team  Leader 

* 

6.  The  Principal 

Managers 

Should  be  a  cohesive 

rvrv 

Design  Team 

Analysts 

group  each  with  de- 

^**</** 

Members 

Engineers 

signated  areas/ 

User  Representatives 

responsibilities. 

7.  Sponsors 


8.  Administration 


Higher  Level  Manager 
Former  User 


Budget 

Procurement 


Can  be  regarded  as 
customer,  but  may  not 
be  the  actual  user. 


May  only  be  concerned 
with  procurement,  schedules, 
and  the  budget  process. 
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TABLE  11.  Steps  to  be  Taken  in  Producing  an 
Overview  Description. 

1.  Define  and  Document  the  Objective. 

2.  Define  and  Document  Special  Requirements. 

3.  Define  and  Document  the  System  (broken  into  Major  System  Components). 

4.  Define  and  Document  the  Operator(s)  Characteristics. 

5.  Identify  and  Document  the  Operating  Times,  Places,  and  Environmental 
Conditions  (see  Table  3). 

6.  Identify  and  Document  Other  Factors  or  Entities  Affecting  System 
Operation,  (See  Table  3). 

7.  Describe  and  Document  the  Operating  Concepts. 

a.  How  will  the  system  be  used? 

b.  What  role  do  major  system  components  play? 

8.  Determine  General  System  Measures  of  Effectiveness. 
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TABLE  12.  Brief  Example  of  an  Overview  Description  for  a  Transport  System. 


1.  Objective:  Transport  people  (the  public)  via  surface  bus  from  place  to 
place  within  a  city. 

2.  Special  Requirements:  The  system  should  have  the  capability  of 
transporting  individuals  on  foot,  individuals  with  their  bicycles,  individuals 
on  crutches  and  canes,  and  individuals  in  wheel  chairs.  Pets  need  not  be 
transported. 

3.  Major  Components:  The  basic  system  is  considered  to  be: 

a.  The  bus  (vehicle)  and  accessories  (e.g.,  loading  ramp) 

b.  The  driver 

c.  The  conductor/subsystem  operator  (if  required) 

d.  A  centralized  dispatcher 

e.  The  communications  system  between  bus  and  dispatcher. 

4.  Operator  Characteristics:  The  operator  characteristics  (driver/conductor) 
are: 

a.  Adult  male  or  female 

b.  90  percent  of  general  population  with  respect  to  height,  weight, 
strength  characteristics 

c.  Possesses  chauffeur's  driving  license  for  bus-type  vehicles. 

5.  Operating  Conditions: 

a.  24  hrs  per  day,  7  days  per  week 

b.  Anywhere  within  city  limits,  including  on  freeways  (alleys  and  such 
narrow  passageways  excepted). 

c.  Operation  is  required  in  snow,  rain,  fog  (restricted  visibility),  daylight, 
dark;  dry  and  wet  or  snow/ice-covered  pavements  are  included. 

d.  Operation  must  also  be  in  light  to  heavy  city  traffic;  maximum  street 
slope  is  5%  grade. 

6.  Other  Factor/Entities:  Other  similar  vehicles  will  be  performing  same  or 
parallel  functions;  street  signs,  traffic  regulations,  controls,  and  advisories; 
passenger  characteristics  and  density;  type/shape  of  bus  route. 

7.  Operating  Concepts:  Passengers  will  be  picked  up  and  dropped  off  at 
designated  locations;  some  schedule  that  passengers  will  know  ahead  of  time 
will  be  established.  Bus  accessories  will  load  passengers;  bus  will  transport 
passengers;  other  operating  conc.-pts  should  be  determined  without 
necessarily  using  traditional  methods. 

8.  MOE:  Percent  of  passengers  desiring  transport  that  can  be  transported  in 
normal  and  peak  hours;  timeliness  in  keeping  schedule;  accident  rate; 
customer  satisfaction  (see  Table  1). 


c.  .V  .*• 
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First  Review 


When  the  overview  description  (Table  11)  is  complete,  it  should  be  reviewed 
by  two  groups  shown  in  Table  10:  #6,  principal  design  team  members  and  #7, 
sponsors.  Understanding  and  approval  by  these  two  groups  will  help  insure:  (1) 
that  everyone  will  be  working  to  the  same  goals  within  the  same  framework,  and 
(2)  that  the  customer  will  get  what  he  thinks  he  is  buying. 

The  analysts  circulating  the  overview  description  must  expect  that  changes 
will  be  made  in  it.  The  changes  may  even  be  improvements  in  some  cases!  It  is 
rare  that  changes  will  not  be  made  (at  a  minimum,  wording  or  preferential 
editing). 


Second  Review 


The  changed  Overview  Description  should  next  be  reviewed  by  a  few  current 
users  of  a  similar  system  (// 4  in  Table  10).  The  analyst  should  present  to,  talk 
with,  or  interview  bus  drivers,  pilots,  control  tower  operators,  customers,  or 
whoever  is  appropriate.  Do  the  concepts  in  the  description  make  sense  in  light 
of  their  experience?  Was  anything  left  out? 

It  must  be  kept  in  mind  that  many  users  are  not  familiar  with  the  design 
process,  or  with  advanced  concepts  in  technology.  The  presentation  of  material 
to  be  reviewed  should  orient  them  toward  thinking  about  an  advanced  system. 
Their  comments  must  be  interpreted  in  this  light  also. 


SYSTEM  DESCRIPTION 


Hardware/Software 


The  system  must  now  be  described  in  more  detail,  with  major  components 
that  are  thought  to  relate  to  effectiveness  clearly  identified.  The  completeness 
of  the  description  will  be  related  to  the  stage  of  design/development  and  to  the 
conventionality  of  the  system.  The  newer  system  will  have  a  more  complicated 
description  since  alternate  components  and  configurations  will  have  to  be 
included.  Table  13  lists  the  steps  to  be  taken  in  this  system  description. 

Table  14  shows  a  partial  description  of  the  vehicle  of  the  transport  system 
described  earlier.  Table  14  is  only  an  example,  but  it  does  indicate  that  some 
mission  requirements  and  real  world  constraints  will  lead  to  an  initial  set  of 
numbers.  Some  numbers  may  be  engineers'  first  guesses  that  are  given  to  the 
human  factors  engineer  or  analyst.  The  numbers  can  be  a  first  iteration  in  the 
design  process  and  as  such  are  "negotiable."  Other  items  like  the 
communications  sub-system  would  have  to  be  described  in  a  similiar  way. 
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TABLE  13.  Steps  in  System  Description. 


1.  Use  information  describing  the  system  that  already  exists. 

2.  Include  only  those  components  that  are  thought  to  relate  to  overall  system 
effectiveness.  Keep  this  first  description  general;  don't  describe  the 
detailed  parts  of  a  component. 

3.  Identify  design  decisions  yet  to  be  made  on  specific  components.  That  is, 
indicate  which  components  are  still  selectable,  and  indicate  the  range  in 
characteristics  that  is  still  possible. 

4.  Include  alternative  configurations  if  a  specific  one  has  not  yet  been  adopted. 

5.  Have  the  system  description  reviewed  by  the  design  team. 


TABLE  14.  Bus  Description. 


1.  Inside  Dimensions:  7  feet  wide  by  7  feet  high  by  25  feet  long. 

2.  Maximum  Outside  Dimensions:  8  feet  wide  by  29  feet  long. 

3.  Will  contain  28  to  36  seats  in  normal  configuration;  exact  number  to  be 
determined  (TBD). 

4.  Will  contain  2  to  4  spaces  for  wheelchairs;  exact  number  TBD. 

5.  Will  contain  2  to  4  seats  for  handicapped/disabled;  exact  number  TBD. 

6.  Will  have  two  or  three  loading/unloading  doors.  Door  location  will  be 
right  front  and  right  side  of  bus.  Seat/space  layout  TBD,  but  will  be 
compatible  with  doors. 

7.  Vehicle  will  use  standard  bus  wheels  and  tires  with  minimum  vertical 
clearance  of  1  foot  between  tires. 

8.  Will  have  mechanism  for  conveniently  loading/unloading  wheelchairs/ 
handicapped,  operable  by  driver.  Mechanism  type  TBD  (ramp,  lift). 
The  time  required  to  load/unload  1  wheelchair  is  estimated  by  engineers 
to  be  45  to  90  seconds. 

9.  Vehicle  can  be  driven  up  to  65  mph. 

10.  Vehicle  will  have  3  rear-view  mirrors  for  driver,  and  can  have  1  TV 
monitor  with  up  to  2  selectable  cameras  inside  or  outside,  if  required. 
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Table  14  contains  items  that  may  not  be  directly  relevant  to  the 
effectiveness  analysis  (e.g.  clearance),  and  items  yet  to  be  determined  (TBD). 
To  follow  the  steps  in  Table  13,  the  analysts  or  design  engineers  must  propose 
alternate  configurations  accounting  for  the  items  TBD.  One  could  hypothesize 
at  least  8  reasonable  configurations  from  the  data  given  in  Table  14. 


Third  Review 

If  these  configurations  are  suggested  by  the  analysts,  they  must  be  reviewed 
by  the  managers  and  design  team  members  for  acceptability  at  this  point  in  the 
design  process.  This  review  process  is  the  third  iteration  of  clarifying  assumed 
system  characteristics. 


Mission  Description 

The  functions  that  must  be  performed  with  the  system  must  be  described  for 
one  complete  assignment,  or  mission.  If  the  system  will  be  used  on  different 
kinds  of  missions,  each  should  be  described.  If  different  configurations  require 
different  functions,  they  must  be  included  also. 

Standard  human  factors  analysis  techniques  include  function  and  task 
description,  operation  sequences,  and  construction  of  mission  profiles  and 
23  24  25  27  28 

timelines.  ’  ’  ’  ’  If  this  work  is  complete,  it  can  be  used  by  the  analysts; 
if  not,  they  must  do  at  least  the  top  level  (more  general)  work  themselves.  A 
good  deal  of  other  human  factors  work  like  workspace  layout  and  seat  design  is 
also  required,  but  need  not  be  included  in  this  type  of  analysis.  An  excellent 
illustration  of  this  other  work  (on  our  bus  example)  was  discovered  in  a  London 

bookshop  after  this  report  was  written.^ 

Table  15  gives  some  comments  on  this  mission  description  process,  and 
Table  16  shows  a  very  brief  listing  using  our  bus  example. 


27 

Air  Force  Systems  Command.  Human  Engineering  Procedures  Guide,  by 
Charles  W.  Geer,  Boeing  Aerospace  Company,  September  1981,  AFAMRL- 
TR-81-35,  Wright- Patterson  Air  Force  Base,  Ohio  (UNCLASSIFIED). 

28 

Naval  Air  Systems  Command,  System  Design  Handbook  for  Advanced  Patrol 
Aircraft.  (NAVAIR  03PA2),  3uly  1973,  prepared  under  contract 
N00019-71-C-0545.  (Boeing  Aerospace  Company,  Seattle,  Report  No. 
D180- 15045-1,  publication  UNCLASSIFIED). 

29 

*?Brooks,  B.  M.,  "Bus  Design:  A  Study  of  Passenger  Capabilities  and 
Requirements"  in  Design  for  Work  and  Use:  Case  Studies  in  Ergonomics  Practice. 
Volume  2,  edited  by  H.  G.  Maule  and  3.  S.  Weiner,  1981,  Taylor  and  Francis  Ltd, 
London. 
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TABLE  15.  Notes  on  Mission  Description. 


1.  List  and  briefly  describe  functions  that  must  be  performed  to  accomplish 
the  mission. 

2.  List  and  briefly  describe  the  major  tasks  required  in  each  function.  A 
function  and  task  allocation  is  implied  here.  It  should  have  been  done 
elsewhere  as  part  of  the  human  factors  work  on  the  program.  It  may  have 
been  done  implicitly  by  the  design  engineers. 

3.  Do  the  above  (1  and  2)  for  each  configuration  and  mission,  or  show  branches 
in  the  appropriate  mission  segments  to  include  the  alternatives. 

4.  Construct  mission  profiles  for  illustrating  the  concepts  to  reviewers.  Those 
for  aircraft  missions  sometimes  show  the  flight  path  plotted  by  altitude  and 
ground-track  with  functions  or  tasks  indicated.  Others  can  show  a  timeline, 
with  the  function  or  tasks  shown  in  the  proper  sequence. 

5.  Note  assumptions,  alternatives,  questions,  or  issues  that  come  up  in  doing 
this  work.  This  side  information  may  prove  helpful  later,  or  must  be 
resolved  if  controversial.  A  systematic  procedure  should  be  established  to 
keep  a  record  of  changes  in  the  missions  as  the  development  of  the  system 
progresses. 

6.  Have  completed  missions  reviewed  by  design  team  and  users. 
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TABLE  16.  Functions  and  Tasks  for  Bus  System. 


Function 


Loader/Unloader  Device  for  Handicapped  Passengers 
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Operating  Conditions 

A  more  complete  description  of  the  operating  and/or  environmental 
conditions  than  that  given  in  item  //5,  Table  12  will  be  required.  The  analysts 
should  list  the  conditions  for  both  operators  and  machines.  Guidelines  such  as 
Tables  7  and  S  should  be  used;  a  brief  example  is  shown  in  Table  17. 

This  tabulation  should  be  reviewed  by  some  members  of  the  users  group.  Is 
the  tabulation  accurate?  Have  some  items  been  omitted?  Additional  insight  can 
be  gained  by  attaching  a  questionnaire  to  Table  17,  asking  for  relative  difficulty 
ratings. 


TABLE  17.  Some  Operating  Conditions  for  Bus  Example. 


Condition  Effect  on  Performance 


1.  Street  Surface 

a.  Dry,  good  repair 

b.  Wet 

c.  Icy /snowy 

d.  Under  repair 

2.  Weather 

a.  Dry,  calm 

b.  Rain,  snow 


c.  Wind 

d.  High  or  low  temperatures 

3.  Traffic  Density  (Time  of  Day) 

a.  Daytime,  non-rush  hour 

b.  Rush  hours 

7-9,  4-6  weekdays 

c.  Nighttime 


d.  Weekends 

4.  Passenger  Density 

5.  Street  Slope  (Grade) 

a.  Flat 

b.  Hilly 


a.  Nominal  case 

b.  Slows  bus/traffic 

c.  Slows  bus/traffic 
Slows  loading/unloading 

d.  Slows  bus/traffic 


a.  Nominal  case 

b.  Slows  bus/traffic 
Restricts  visibility 
Slows  loading/unloading 
Slows  fare  collection 

c.  Could  change  fuel  consumption 

d.  Could  change  fuel  consumption 


a.  Nominal  case 

b.  Slows  driving 

Increases  number  of  passengers 

c.  Faster  loading/unloading 
than  nominal  case 

Slower  driving  than  nominal 
case 

d.  Similar  to  nominal  case 
Assumed  same  as  above 


a.  Nominal  case 

b.  Slows  driving 
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Measures  of  Effectiveness 


Measures  of  effectiveness  can  now  be  generated  using  the  descriptions  of 
the  system  and  missions.  Guidelines  in  the  previous  MOE  section  and  Appendix  B 
will  be  helpful.  The  MOEs  should  be  chosen  so  that  they  can  be  used  in 
answering  questions  or  making  trade-offs  that  always  come  up  in  the  design 
decision  process.  Hence,  they  should  be  related  to  the  system  description  (Table 
14).  Some  MOEs  may  have  already  been  suggested  by  design  engineers  before 
the  analysis  was  begun.  The  MOEs  also  should  be  related  to  the  functions  and 
tasks  (Table  15  and  16).  Two  example  MOEs  are  shown  in  Table  18  with  some 
amplifying  (or  complicating)  comments.  These  MOEs  will  have  to  be  described  in 
more  detail,  quantitatively,  later  in  the  analysis  process.  More  about  that  later. 


TABLE  18.  Two  Possible  MOEs  for  Bus  Example. 


1.  Timeliness  in  maintaining  schedule  along  route. 

a.  Routes  must  be  defined. 

b.  Schedules  must  be  defined. 

c.  Routes  and  schedules  could  be  varied  with  time  of  day  or  week  (this  is  a 
study  in  itself). 

d.  Number  of  busses  assigned  to  one  route  as  a  function  of  time  of  day  or 
week  will  affect  timeliness. 

2.  Percent  of  people  desiring  transportation  at  any  one  time  that  can  be 

transported. 

a.  Percent  can  be  affected  by  choice  of  route,  schedule,  and  number  of 
busses  assigned  to  a  route. 

b.  Data  is  required  on  number  of  passengers  that  can  stand  on  bus  if  seats 
not  available.  This  may  also  be  a  policy  decision. 


Fourth  Review 


At  this  point  in  the  analysis,  Boxes  1  through  4  in  Figure  2  have  been 
addressed.  The  analysts  have  assembled  the  necessary  background  information 
and  are  ready  to  start  construction  of  an  algorithm  for  calculating  system 
effectiveness.  It  would  be  wise  to  have  this  baseline  information  approved  by 
the  program  managers,  principal  design  team  members,  and  also  by  the  sponsors 
if  possible.  It  is  necessary  to  get  agreement  that,  (1)  this  is  the  system,  (2)  this 
is  how  it  will  be  used,  and  (3)  this  is  how  it  will  be  judged. 
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Action  Items 


A  number  of  comments  and  questions  will  have  occurred  to  the  analysts 
throughout  the  description  process  (e.g.  item  5  in  Table  15).  These  should  be 
formally  recorded  and  included  in  the  planning  and  analysis.  The  items  can  be 
grouped  by  their  nature;  some  from  the  bus  example  are  shown  in  Table  19. 

The  nature  of  these  items  depends  upon  the  stage  at  which  the  analysis  is 
performed.  If  one  is  estimating  the  effectiveness  of  a  proposed,  or  hypothetical 
system,  many  questions  concerning  system  characteristics  will  come  up.  These 
will  have  already  been  answered  for  an  existing  system. 


MODELING  THE  SYSTEM 


Functions/Tasks  To  Be  Modeled 


Only  those  functions,  tasks,  and  components  need  to  be  modeled  that  affect 
system  performance  and  effectiveness.  All  functions,  tasks,  and  components  are 
essential  to  the  system  (or  they  wouldn't  be  there),  but  some  can  be  assumed  to 
pose  no  problem  in  system  operation. 

The  listing  of  functions  and  tasks  (Table  16  in  our  example)  can  be  used  to 
determine  what  should  be  modeled.  The  steps  to  be  taken  are  shown  in  Table  20 
with  brief  examples  from  the  bus  design  shown  in  Tables  21  and  22. 

Column  B  of  Table  22  in  the  example  would  be  expanded  to  more  detail  in  an 
actual  analysis.  That  expanded  description  will  be  used  as  a  basis  for 
performance  modeling.  A  brief  example  is  shown  in  Table  23. 


Conditions  To  Be  Modeled  or  Described 


The  conditions  to  be  modeled  or  described  are  also  indicated  in  Tables  22 
and  23.  The  type  of  specific  descriptions  must  be  related  to  the  use  of  the  data 
in  the  model  or  algorithm,  and  will  be  discussed  later  in  this  report. 


)uantifi  nation  of  MOEs 


The  next  task  is  to  quantify  the  MOEs,  being  very  specific,  such  that  they 
are  related  to  the  mission  (functions  and  tasks),  the  system  being  designed 
(equipment  components)  and  the  conditions  of  operation.  A  MOE  may  be  a  single 
item,  but  it  is  usually  made  up  of  a  number  of  components.  For  example,  the 
time  required  to  load  a  bicycle  may  be  made  up  of  the  time  required  to  (1)  open 
the  rack,  (2)  lift  the  bicycle  onto  the  rack,  and  (3)  close  the  rack. 


, 


r 
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TABLE  19.  Action  Items/Notes. 


1.  Items  To  Be  Determined  (TBDs) 

a.  Number  of  doors  in  bus 

b.  Number  of  wheelchair  spaces 

c.  Number  of  handicapped  seating  spaces 

d.  Number  of  regular  passenger  spaces 

e.  Number  of  passengers  allowed  to  stand 

f.  Number  of  bicycle  spaces  available. 

2.  Supporting  Analysis  Required 

a.  Generate  feasible  alternate  configurations 

b.  Conduct  man/machine  function  allocation 

c.  Conduct  seat/space/door  layouts  to  validate  feasibility  of  con¬ 
figurations 

d.  Generate  representative  routes 

e.  Generate  representative  (nominal)  schedule. 

3.  Data  Required 

a.  Traffic  flow  (speed)  versus  driving  conditions  and  time  of  day 

b.  Bus  speeds  versus  driving  conditions  and  street  conditions 

c.  Operating  times  of  loader/unloader 

d.  Loading/unloading  times  of  bicycles 

e.  Loading/unloading  times  of  pedestrians 

f.  Number  of  people  desiring  transportation  as  a  function  of  time  of  day 
and  week. 

4.  Bothersome  Questions 

a.  How  can  we  address  the  interactions  between  route,  timeliness, 
time-of-day,  and  number  of  busses  assigned? 

b.  Must  we  do  a  complete  queueing  study  which  would  include  various  bus 
configurations? 

c.  How  do  we  establish  "representative"  routes  and  schedules? 

5.  Assumptions  Made 

a.  Human  factors  techniques  will  be  used  to  produce  bus  layout  (seats, 
doors,  spaces),  as  given  in  reference  29. 

b.  Human  factors  techniques  will  be  used  to  determine  passenger  flow 
through  bus  and  fare  collection  options. 

c.  Only  driver  is  included  in  effectiveness  analysis  at  this  point;  if 
conductor  is  required  by  (a)  and  (b)  above,  provision  will  be  made  later 
in  the  anlaysis. 


t- 
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TABLE  20.  Steps  in  Selecting  Items  to  be  Modeled. 


1.  Augment  the  list  of  functions  and  tasks  with  comments,  remarks,  or 
observations  to  provide  a  more  complete  description  of  the  items. 

2.  Estimate  the  strength  of  the  effect  on  the  MOEs  of  each  of  the  tasks. 
Include  secondary  effects. 

3.  Indicate  which  tasks  are  affected  by  the  operating  conditions  listed  earlier. 
Pair  up  specific  tasks  with  the  specific  operating  conditions. 

4.  Make  a  preliminary  decision  as  to  what  parts  of  the  system  and  system 
operation  should  be  modeled. 

5.  Document  the  above. 


TABLE  21.  Comments  on  Functions  and  Tasks  (see  item  f/1,  Table  20). 


FUNCTIONS  &  TASKS  COMMENTS 


1.  Preparation 

a.  Check  Oil 

b.  Check  Tires 

c.  Check  Doors  &  Loaders 

d.  Check  Vehicle  Lights 

e.  Fill  Fuel 

f.  Get  Route  Assignment 


2.  Transport  Bus  to  First 
Bus  Stop 

a.  Monitor  Engine 

b.  Monitor  Traffic 

c.  Drive  Bus 


These  items  are  "standard"  procedures,  with 
experience  to  draw  upon  from  the  past. 
Vehicle  should  be  designed  such  that 
maintenance  checks  can  be  made  easily. 
These  items  are  not  time-critical. 

These  items  might  be  performed  by  main¬ 
tenance  personnel,  not  bus  crew. 

Existing  systems  can  provide  data. 


"Standard"  driving  procedures  are  employed. 
There  is  experience  from  the  past  to  draw 
upon.  Will  bus  make  it  similar 
to  older  (existing)  busses  in  "drivability"? 

If  so,  nothing  is  really  new  here.  Existing 
systems  can  provide  data. 


NWC  TP  6541 


» 


TABLE  21.  Comments  on  Functions  and  Tasks.  (Continued) 

Functions  and  Tasks  Comments 

3.  Load  Passengers 

a.  Park  at  Bus  Stop 

b.  Open  Door(s) 

c.  Operate  Loader(s) 

d.  Load  Bicycles 

e.  Load  Other  Passengers 

f.  Sell  Tickets/Collect  Fares 

g.  Close  Door(s) 

h.  Secure  Loader(s) 

i.  Check  Passengers' 

Positions 

4.  Transport  Passengers  to 
Next  Bus  Stop 

a.  Monitor  Engine  Instruments  Part  of  this  is  the  same  as  #2,  above. 

b.  Monitor  Traffic  We  must  establish  the  requirement  for, 

c.  Drive  Bus  and  type  of  communication  with  dispatcher. 

d.  Monitor  Passengers  Larger,  transportation  system  operation 

e.  Answer  Passengers'  study  should  be  conducted.  If  not,  as- 

Questions  sumptions  as  to  communication  must  be 

f.  Communicate  with  made.  Can  passengers'  questions  be 

Dispatcher  answered  with  the  aid  of  some  subsystem? 

Can  such  aids  be  suggested/recommended? 

5.  Unload  Passengers 

a.  Park  at  Bus  Stop  Same  as  #3  above.  These  items  may  be 

b.  Open  Doors  performed  concurrently  with  //6  below. 

c.  Unload  Passengers  Bus  layout  and  design  configuration  should 

d.  Operate  Loader(s)  address  this  possibility. 

(to  Unload) 

e.  Unload  Bicycles 

6.  Load  Passengers 

a.  Operate  Loader(s) 

b.  Load  Bicycles 

c.  Load  Passengers 

d.  Sell  Tickets/Collect 
Fares 

e.  Close  Doors 

f.  Secure  Loader(s) 

g.  Check  Passengers' 

Positions 
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Same  as  #3  above.  These  items  may  be 
performed  concurrently  with  #5  above. 
Bus  layout  and  design  configuration 
should  address  this  possibility. 


These  items  are  a  function  of  bus  design 
(configuration)  and  bus  crew  size  (one  or 
two).  Loader  characteristics  must  be 
established.  Who  loads  bicycles  (bus 
crew,  or  riders)?  How  is  fare  collected? 
Operating  times  of  doors  and  loader(s) 
must  be  obtained  or  estimated.  AH 
functions  here  are  described  by  capacity 
and  time  required. 
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TABLE  22.  Modeling  Checklist. 


Functions  and  Tasks 


Effect 
on  MOEs 
(Table  18) 


Affected 

by 

Conditions 
(Table  17) 


Items 
to  be 
Modeled 


1. 

Preparation 

a.  Check  Oil 

Low 

b.  Check  Tires 

Low 

c.  Check  Doors  <3c  Loaders 

Low 

d.  Check  Vehicle  Lights 

Low 

e.  Fill  Fuel 

Low 

f.  Get  Route  Assignment 

Low 

2. 

Transport  Bus  to  First 

Bus  Stop 

a.  Monitor  Engine  Instruments 

Low 

b.  Monitor  Traffic 

Low 

c.  Drive  Bus 

Low 

3. 

Load  Passengers 

a.  Park  at  Bus  Stop 

Low 

b.  Open  Door(s) 

Low 

c.  Operate  Loader(s) 

High 

d.  Load  Bicycles 

High 

e.  Load  Other  Passengers 

High 

f.  Sell  Tickets/Collect  Fares 

Medium 

g.  Close  Door(s) 

Low 

h.  Secure  Loader(s) 

Low 

i.  Check  Passengers'  Positions 

Low 

4. 

Transport  Passengers 

Next  Bus  Stop 

a.  Monitor  Engine  Instruments 

Low 

b.  Monitor  Traffic 

Low 

c.  Drive  Bus 

High 

d.  Monitor  Passengers 

Low 

e.  Answer  Passengers'  Questions 

Low 

f.  Communicate  with  Dispatcher 

Low 

NWC  TP  6541 


TABLE  22.  Modeling  Checklist.  (Continued) 


A 

B 

C 

Functions  and  Tasks 

Effect 
on  MOEs 
(Table  18) 

Affected 

by 

Conditions 
(Table  17) 

Items 
to  be 

Modeled 

5. 

Unload  Passengers 

a.  Park  at  Bus  Stop 

Low 

No 

Yes 

b.  Open  Doors 

Low 

No 

Yes 

c.  Unload  Passengers 

High 

Yes 

Yes 

d.  Operate  Loader(s)  (to  Unload) 

High 

Yes 

Yes 

e.  Unload  Bicycles 

High 

Yes 

Yes 

6. 

Load  Passengers 

a.  Operate  Loader(s) 

High 

Yes 

Yes 

b.  Load  Bicycles 

High 

Yes 

Yes 

c.  Load  Passengers 

High 

Yes 

Yes 

d.  Sell  Tickets/Collect  Fares 

Medium 

No 

Yes 

e.  Close  Doors 

Low 

No 

Yes 

f.  Secure  Loader(s) 

Low 

No 

Yes 

g.  Check  Passengers'  Positions 

Low 

No 

Yes 
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TABLE  23.  Expansion  of  Item  5  in  Table  22 


Conditions 
Affecting 
Tasks 


Nature 

of 

Effect 


Remarks 


5.  Unload  Passengers 

a.  Park  at 

Bus  Stop 

Traffic 

Density 

Minimal  effect; 
high  density  may  slow 
parking  somewhat 

Data  from 
existing  systems 
should  apply 

b.  Open  Doors 

None 

None 

None 

c.  Unload 
Passengers 

Wet  street 

Rain 

Snow 

Snowy/icy  streets 

Each  passenger 
disembarks  slower  than 
under  good  conditions 

Data  from 
existing 
systems 
should  apply 

d.  Operate 
Loader 

Hilly 

Rain 

Wet  street 

Snowy /icy  streets 

Slows  operation  of 
loader.  Each 
passenger  disembarks 
slower 

Data  needed 
from  simulator 
or  submodel 

e.  Unload 
Bicycles 

Snow 

Rain 

Snowy /icy  streets 

Slows  each  unloading 
operation 

Bicycles  may  be 
unloaded  simul¬ 
taneously  or 
serially  by 
riders.  Data 
needed  from 
existing  system 
or  simulation. 

A  MOE  component  is  often  a  measure  of  performance  of  an  equipment 
component  and/or  its  human  operator.  To  illustrate  the  concept,  we  can  create 
a  couple  of  acronyms  and  write  a  couple  of  equations. 

MOEC  =  MOP, 

where  MOEC  is  the  MOE  component  and  MOP  is  the  measure  of  performance  of 
the  equipment  or  operator.  Also,  it  may  be  that 

MOE  =  J^MOEC  =2>OP. 
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The  way  the  MOEC  are  combined  (or  summed)  will  vary  with  the  nature  of  the 
MOE  and  MOEC.  The  reader  should  remember  that  simple  addition  of  MOE 
components  may  not  be  representative  of  the  actual  situation.  Hence,  breaking 
down  MOE  into  components  is  often  a  complex  and  challenging  task,  and 
deserves  special  attention. 

The  task  of  quantifying  an  MOE  includes  breaking  it  into  its  components, 
relating  each  to  the  system  and  mission,  and  showing  how  the  components  can  be 
put  back  together  again,  analytically  speaking.  The  steps  to  be  taken  in  this 
process  are  shown  in  Table  24.  Item  #2  in  Table  24  is  illustrated  with  an 
example  from  the  bus  design  case  in  Table  25. 


TABLE  24.  Steps  in  MOE  Quantification. 


1.  Start  with  the  MOEs  listed  and  agreed  upon  earlier  (Table  18  in  our 
example). 

2.  Break  each  MOE  into  components,  if  possible,  such  that  each  can  be  related 
to  the  functions  and  tasks  listed  earlier.  Include  only  those  items  to  be 
modeled  (Column  C,  Table  22  in  our  example). 

3.  If  the  MOE  cannot  be  simply  broken  into  components  as  in  (2),  develop  the 
special  definitions  required  to  translate  the  system  (or  subsystem) 
performance  into  the  appropriate  MOE.  These  definitions  can  usually  take 
the  form  of  mathematical  statements.  Document  the  assumptions  made  in 
this  process. 

4.  Indicate  which  equipment  component(s)  and/or  human  operator(s)  are  related 
to  each  MOE  component.  (The  MOE  component  is  really  a  measure  of 
performance). 

5.  Indicate  (qualitatively)  how  design  decisions  might  be  affected  by  the  MOEs. 
For  example,  the  passenger-loader  with  the  shortest  operating  time  would 
be  preferred  with  everything  else  equal.  (This  item  is  an  "extra"  that 
reminds  us  why  we  are  doing  the  analysis  in  the  first  place.) 

6.  Verify  that  the  MOE  components  are  compatible  with  any  lower-level  MOE 
that  may  be  used  in  analysis  or  testing. 

7.  Show  how  the  MOE  components,  or  special  MOE  definitions,  can  be 
"reassembled"  to  form  the  top-level  MOE  first  described  (Table  18).  A 
block  diagram  may  be  useful  in  this  process. 

8.  Have  MOEs  that  are  formulated  in  item  (3)  above  reviewed  by  any  other 
analysts  on  the  program  and  principal  design  team  members  (//l  and  6,  Table 
10). 
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TABLE  25.  Example  of  MOE  Steps  for  Item  5,  Table  22. 


Function 

MOE 

5.  Unload  Passengers 

Total  time  from  bus  being  within  30 
feet  of  bus  stop  to  when  everything 
and  everyone  who  wishes  to  has  left 
the  bus  (Tj  in  seconds). 

Tasks 

MOEC 

b.  Open  Doors 


a.  Park  at  Bus  Stop  Time  required  from  being  within  30 

feet  of  Bus  stop  to  coming  to  full  stop 
(tj  in  seconds). 

b.  Open  Doors  Time  from  coming  to  full  stop  to  when 

doors  are  fully  open  so  that  passengers 
can  get  out  in  seconds). 

c.  Unload  Passengers  Time  from  when  doors  are  fully  open 

(except  handicapped  passenger  loader)  until  last  passenger  has  left  and 

cleared  the  bus  (t^  in  seconds).  This 
is  a  function  o  i  the  number  of 
disembarking  passengers  and  number 
of  doors. 


d.  Operate  Loader 


e.  Unload  Bicycles 


Time  from  when  bus  comes  to  full  stop 
to  when  all  loader -passengers  have 
been  unloaded  (t^  in  seconds). 

Time  from  when  bus  comes  to  full  stop 
to  when  all  bicycles  have  been  un¬ 
loaded  (t*  in  seconds).  This  is  a 
function  of  bicycle  carrier  char¬ 
acteristics,  number  of  bicycles,  and 
how  they  are  unloaded. 


NOTE:  tj  +  t2  +  tj  =  time  to  unload  walking  passengers 
t,.  =  time  to  unload  handicaDDed  Dassenee 


time  to  unload  handicapped  passengers;  could  be  in  parallel 
with  tj  +  +  ty 

time  to  unload  all  bicycles;  could  be  in  parallel  with  tj  +  t2  + 


Tj  =  some  combination  (or  largest  of)  the  above. 
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Item  #3  in  Table  24  can  be  a  very  complicated  and  challenging  process;  it  is 
not  always  straightforward.  To  illustrate,  the  MOEC  shown  in  Table  25  are  all 
times  that  can  be  easily  combined  to  form  the  MOE  for  item  //5.  That  MOE,  in 
turn,  can  be  combined  with  other  times  to  get  an  estimate  of  the  total  time 
required  for  the  same  number  of  functions  and  tasks.  But  what  is  "timeliness"  in 
Table  18?  Can  the  MOE  be  modified  t-»  avoid  defining  routes,  schedules,  and 
numbers  of  busses  assigned? 


Modification  of  MOEs 


At  this  point  in  the  analysis  it  may  be  advisable  to  modify  the  MOEs  to 
incorporate  or  reflect  new  information.  Table  26  gives  some  comments  on  the 
pros  and  cons  of  MOE  modification.  Any  modifications  must  be  approved  by  the 
same  people  who  reviewed  the  original  MOEs. 


TABLE  26.  Advantages  and  Disadvantages  in  Modifying  MOEs. 


Advantages 

1.  Modification  may  be  required  as  indicated  by  a  more  detailed  look  at  the 
system  and  its  operation.  The  original  MOEs  may  simply  not  fit  the 
situation. 

2.  Modification  may  better  match  the  MOE  components  to  the  equipment  and 
operator  performance. 

3.  Modification  may  save  some  work  (e.g.  specifically  describing  a  mission  or 
scenario). 

Disadvantages 

1.  Modification  may  not  be  acceptable  to  the  sponsor  (customer).  It  simply 
cannot  be  done  if  that  is  the  case. 

2.  Modification  will  require  another  iteration  in  all  work  done  to  this  point  to 
insure  compatibility. 

Requirement 

1.  The  modified  MOEs  must  still  serve  the  desired  purposes  (e.g.  provide 
information  for  use  in  making  design  decisions). 

2.  If  the  modification  to  the  MOEs  is  a  simplification,  the  new  MOEs  should  be 
useful  in  constructing  the  original  MOEs  if  more  time  (resources)  become 
available  for  analysis. 
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As  an  example,  a  modification  of  the  MOE  given  in  item  1,  Table  18  could 
be: 

The  total  time  required  to  complete  one  "cycle"  in  the  bus 
mission,  i.e.  items  4,  5,  and  6  in  Table  22. 

This  MOE  could  be  used  to  compare  component  design  requirements, 
configurations,  and  operator  performance  under  various  operating  conditions. 
Routes,  schedules,  and  number  of  busses  assigned  to  a  given  route  need  not  be 
determined.  The  MOE  could  be  accumulated  to  reflect  a  higher-level  MOE:  time 
to  cover  an  entire  route. 

Another  example  of  a  MOE  modification  that  might  be  pointed  out  is  to 
severely  downgrade  item  //2  in  Table  18.  The  number  of  people  that  can  be 
carried  on  the  bus  could  be  used  as  a  simple  MOE  of  the  bus  configuration.  What 
mixes  of  sitting,  standing,  wheelchairs,  etc.  are  feasible?  This  MOE  will  interact 
with  the  time  MOE;  it  would  probably  be  good  to  be  able  to  conduct  the  design  to 
maximize  the  number  of  people  and  minimize  handling  time,  with  comfort  and 
safety  requirements  not  violated.  The  number-of-people  MOE  could  then  be 
used  in  a  higher  level  MOE  (percent)  when  the  demand  for  transporatation  has 
been  established  (see  Requirement  //2  in  Table  26).  Downgrading  to  number  of 
people  avoids  the  work  of  having  to  establish  the  customer  demand  just  now  in 
the  analysis. 


Fifth  Review 

MOEs  have  been  modified  as  appropriate,  and  expanded  in  some  detail  as  per 
Table  25.  A  review  and  approval  by  managers,  design  team  members,  and 
sponsors  (#5,  6,  7  in  Table  10)  is  recommended  before  the  effectiveness 
algorithm  is  built. 


BUILDING  THE  ALGORITHM 

An  algorithm  is  a  procedure  for  combining  quantities,  inputs,  or  data  to  get 
the  desired  result.  The  general  concept  is  shown  in  Figure  8.  Equations  or  data 
(or  both)  must  be  combined  in  a  way  that  reflects  the  operating  procedures,  and 
that  produces  the  desired  results:  the  MOE. 

All  of  the  building  blocks  have  been  assembled.  The  tables  that  have  been 
made  up  thus  far  show  the  tasks  to  be  modeled,  the  measures  of  performance  to 
be  modeled,  and  the  conditions  affecting  performance. 

Graphic  representations  of  the  system  and  the  functions  to  be  performed 
may  aid  some  in  formulating  the  mathematics.  A  block  diagram  showing  the 
sequence  of  operations  may  help  (Figure  9).  A  geometric  representation  of 
system  operation  may  be  necessary  to  formulate  the  mathematics.  An  aircraft 
entering  a  traffic  pattern  for  a  landing  would  be  such  an  example  (Figure  10). 
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FIGURE  9.  Examples  of  Block  Diagrams. 
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FIGURE  10.  Example  of  Geometry  Diagram  with  Distances, 
Velocities,  and  Initiation  of  Tasks  Indicated. 


Time  points  could  be  calculated  from  the  distances  and  velocity;  the  time 
available  for  the  various  tasks  can  then  be  derived. 

Figure  11  shows  another  geometric  representation,  where  distance  is  a 
major  parameter.  Within  limits,  time  is  not  a  factor  at  all.  The  measure  of 
effectiveness  is  the  total  number  of  shots  required  to  get  the  ball  from  the  tee 
into  the  hole  on  the  green.  The  direction  and  endpoint  of  a  longer  shot  (D.,  Dj, 
or  D-,)  should  be  planned  as  a  function  of  terrain  and  wind.  The  distance  itself  is 
a  function  of  skill  and  wind. 

A  top-level  diagram  can  be  used  to  show  how  the  MOEs  can  be  calculated 
(Figure  12).  Each  part  of  this  top-level  diagram  can  be  expanded  to  show  its 
components  (Figures  9  and  13).  Figure  13  also  shows  which  conditions  affect 
performance,  and  how  performance  can  be  calculated.  At  this  point,  the  system 
components  and  operating  conditions  will  have  to  be  included  in  the  formulation 
of  the  equations. 
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MOE  =  T  +  OVERLAPPED  COMBINATION  OF  T  AND  Tj 
T  -  TIME  TO  DRIVE  BUS  BETWEEN  STOPS. 


FIGURE  12.  Top-Level  MOE  Diagram. 
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FIGURE  13.  Expansion  of  T.  in  Figure  11 
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The  methods  of  describing  component  and/or  operator  performance  already 
have  been  discussed  in  detail  (see  Figures  3,  4,  5,  and  6).  The  analyst  must  use 
mathematical  formulations,  or  actual  descriptive  data,  whichever  is  most 
appropriate  (and  hopefully  readily  available)  to  estimate  the  performance  of  the 
lowest-level  block  in  the  algorithm  (e.g.,  t^  in  Figure  13).  The  method  of 
combining  the  data  associated  with  the  MOE  components  must  also  be  developed. 


Selection  of  Models 


The  algorithm  can  be  made  up  of  both  mathematical  models  (or  computer 
simulations)  and  empirical  data  (e.g.,  traffic  flow  equations  and  weather 
statistics).  The  types  of  mathematical  formulations  to  use  depend  upon  the 
processes  (operating  proceures)  being  described,  and  the  desired  output. 

A  very  useful  text  by  Martin  gives  definitions  of  different  kinds  of 
models;  an  expansion  of  these  definitions  is  given  below. 

1.  A  deterministic  model  is  an  analytical  representation  of  a 
concept,  system,  or  operation  in  which  there  are  unique  outcomes 
for  a  given  set  of  inputs.  As  an  example,  one  total  time  to 
complete  a  bus  "cycle"  would  be  calculated  from  a  set  of  inputs. 


2.  An  expected  value  model  is  one  in  which  the  expected  values 
(or  means)  are  assigned  to  the  chance  parameters.  Although  in 
common  use,  this  type  of  model  can  lead  to  great  mis¬ 
understanding  by  the  sponsor  or  customer. 

3.  A  stochastic  model  is  one  in  which  the  functional 
relationships  depend  on  chance  parameters  (e.g.,  the  weather 
conditions  over  a  period  of  time).  The  outcomes  for  a  given  set  of 
inputs  can  be  predicted  only  in  a  probabilistic  context.  The 
probability  of  completing  one  cycle  in  less  than  various  times 
would  be  a  result  (P  =  0.95  in  less  than  1  hour,  P  =  0.80  in  less 
than  50  minutes,  etc.).  Monte  Carlo  modeling  is  often  used  to 
make  these  calculations;  that  is,  many,  many  calculations  are 
made  from  random  draws  made  from  assumed  variable  distri¬ 
butions.  These  resultant  calculations  are  then  aggregated  in  some 
way  to  produce  the  final  result. 

A  fourth  model  type  that  has  limited  use  will  be  added  to  Martin's  list. 


4.  A  deterministic/probabilistic  model  is  one  which  produces 
a  unique  outcome  (a  probability  distribution)  from  distributions 
of  input  variables.  Only  one  calculation  is  required  as  opposed 
to  the  thousands  required  in  Monte  Carlo  modeling. 
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Martin,  Francis  F.,  Computer  Modeling  and  Simulation,  John  Wiley  6c  Sons, 
New  York,  1968. 
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An  algorithm  can  use  all  or  some  of  the  features  of  the  four  models  listed 
above.  The  stochastic  or  deterministic/probabilistic  models  are  often  the  most 
representative  of  the  variability  in  the  real  world,  but  getting  valid  input  data 
can  be  difficult. 

The  expected  value  model  can  be  misleading  when  a  number  of  "average" 
performances  are  combined.  This  point  is  important  enough  to  justify  a 
digression. 


Digression:  Using  All  the  Data 

Suppose  three  simple  procedures  as  shown  in  Figure  12  comprise  a  unit, 
where 

MOE  =  Tj  +  T2  +  T3 

Figure  14  shows  two  possible  distribution  sets  for  each  of  the  function  times 
(data  characteristics  were  shown  in  Figures  5  and  6).  In  Set  A,  each  of  the 
functions  could  take  5,  10,  15,  or  20  seconds  to  perform,  with  equal  probability. 
If  one  were  to  average  a  number  of  samples  from  the  performance  time 
distributions,  the  result  would  approach  12.5  seconds.  Realistically,  three 
different  functions  would  have  differently  distributed  times,  however,  as  shown 
in  set  B,  with  different  means  as  shown. 

It  is  assumed  in  this  example  that  the  three  times  are  independant  of  one 
another.  Since  this  is  not  always  the  case,  the  analyst  must  examine  the 
operating  procedures  closely  to  establish  data  independence.  If  there  are 
appreciable  intercorrelations  in  the  data,  these  must  be  accounted  for  in  the 
mathematical  modeling. 

The  "average"  MOE  (T^  +  T ^  +  T^)  for  Set  A  is  37.5  seconds,  and  41.5 
seconds  for  Set  B  (the  means  are  used  to  make  the  calculations). 

Figure  15  shows  the  entire  distribution  of  ail  possible  MOEs  for  both  sets  A 
and  B;  the  MOEs  computed  from  the  means  would  be  exceeded  about  42%  of  the 
time  (1  -  0.58).  If  the  MOEs  are  used  somehow  as  design  requirement  numbers, 
50  to  52  seconds  would  probably  be  better  choices.  People  would  prefer  a  system 
that  does  the  job  95%  of  the  time  to  one  that  works  only  58%  of  the  time. 

Figure  15  illustrates  the  most  complete  form  of  an  MOE,  the  probability 
distribution.  All  of  the  data  (Figure  14)  are  used  to  form  Figure  15.  The 
distributions  in  the  data  are  also  implicitly  included  in  Figure  15.  Although  the 
distributions  shown  in  Figure  14  (A  versus  B)  look  quite  different,  the  two  MOE 
at  the  95%  level  are  50  versus  52  seconds.  The  operational  difference  is 
probably  negligible. 

The  analysis  process  does  not  make  decisions,  but  it  should  provide  the 
decision-maker  with  the  information  he  needs.  Stochastic  models  provide  much 
more  complete  information  than  do  expected  value  models.  The  cumulative 
curve,  as  shown  in  Figure  15,  allows  the  decision-maker  to  see  the  whole  picture. 
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FIGURE  15.  Cumulative  Distribution  of  Ail  Possible  Combinations 
of  Tj  +  T2  +  T,  as  Shown  in  Figure  14.  Components  of  the  sum  are 
weighted  by  the  relative  frequency  of  occurrence  of  each  time. 


Selection  of  Analytic  Tools 

Meanwhile,  back  at  the  ranch,  the  analyst  is  faced  with  selecting  the 
methods,  models,  and  mathematical  tools  to  best  describe  his  problem.  Table  27 
gives  some  characteristics  of  the  models  that  have  been  discussed  earlier.  Table 
28  gives  a  very  superficial  idea  of  the  range  of  analytic  techniques  that  can  be 
used. 
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Deterministic  Model 


a.  A  single  value  of  each  input  is  used  for  any  one  complete  calculation  by 
the  model. 

b.  There  are  no  variations  in  the  model's  output  due  to  chance  elements. 

c.  All  probabilistic  elements  are  non-existent  or  removed  from  the 
problem. 

d.  The  output  of  a  deterministic  model  is  always  the  same  for  a  given  set 
of  inputs. 

e.  Deterministic  model  outputs  can  be  used  in  parametric  studies  where 
the  output  can  be  plotted  as  a  function  of  specific  input  values. 

2.  Expected  Value  Model 

a.  Mean  values  are  assigned  to  parameters  which  actually  vary  somewhat 
by  chance;  zero  variance  is  assigned. 

b.  No  variance  of  the  results  is  determined,  and  no  distribution  function  of 
the  results  is  determined. 

c.  This  model  should  be  used  when  individual  outcomes  do  not  need  to  be 
known,  when  the  model  results  are  not  affected  by  the  variations  of 
individual  inputs,  and/or  when  it  is  not  necessary  to  determine  the 
distribution  function  and  variance  of  the  model  outcomes. 

d.  The  expected  value  criterion  is  useful  in  situations  where  the  long-term 
average  results  are  of  primary  interest. 

3.  Deterministic/Probabilistic  Model 

a.  Some  of  the  inputs  to  the  model  are  distributions  of  variables;  the 
distributions  can  be  either  empirical  (e.g.  Figure  14)  or  analytical. 

b.  One  complete  calculation  by  the  model  includes  all  the  data  in  the  input 
distributions. 


c.  The  output  of  the  model  can  be  probability  distributions  (Figure  15) 
which  reflect  the  naturally  occuring  variability  in  the  inputs.  A  model 
output  is  exact  (i.e.,  not  an  approximation). 
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TABLE  27.  Model  Description.  (Continued) 


d.  The  deterministic/probabilistic  model  can  be  used  when  only  a  few 
input  distributions  are  required,  and  the  formulation  are  relatively 
simple.  Otherwise,  the  computations  get  too  complicated.  It  is  used 
when  data  distributions  (e.g.  weather  data,  human  performance)  are 
required  to  accurately  represent  reality,  and  when  probability 
distributions  in  the  outcome  are  desired. 

Stochastic  Model 

a.  Some  of  the  inputs  to  the  model  are  distributions  of  variables  (empirical 
or  analytical). 

b.  One  complete  calculation  by  the  model  is  made  by  choosing  one  value 
from  each  distribution  for  the  inputs. 

c.  A  very  large  number  (thousands)  of  these  calculations  must  be  made  for 
each  set  of  conditions;  usually  a  Monte  Carlo  simulation  is  used  to 
accomplish  this.  The  results  are  plotted  as  a  distribution  of  outcomes. 
This  distribution  can  be  plotted  cumulatively  (see  Figure  15)  and 
described  statistically  (mean,  standard  deviation).  The  final  result  is  an 
approximation  that  approaches  the  exact  outcome  as  the  number  of 
calculations  for  each  set  of  conditions  increases. 

d.  Stochastic  models  are  used  when  distributions  in  input  data  must  be 
included,  when  a  probability  distribution  for  the  output  is  appropriate, 
and  when  the  model  is  too  complicated  to  compute  by  the  exact 
method.  A  high-speed  computer  is  usually  required. 
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TABLE  28.  Some  Analysis  Techniques. 


1.  Statistical  Description 

a.  A  statistical  description  of  some  sort  is  usually  necessary  to  be  able  to 
describe  the  model  inputs  and  outputs. 

b.  Probability  theory  is  used  in  these  descriptions  and  in  the  formulation 
of  many  models  and  techniques. 

c.  Discrete  Probability  Distributions  and  Continuous  Probability 
Distributions  are  used  in  analytic  and  empirical  studies. 

2.  Linear  Programming 

a.  Linear  programming  is  used  when  the  model  contains  unknown  variables 
represented  by  algebraic  symbols. 

b  Restrictions  or  constraints  in  the  model  can  be  expressed  as  linear 
equations  or  inequalities  which  are  linear  functions  of  the  unknown 
variables. 

c.  The  objective  or  MOE  can  also  be  expressed  as  a  linear  function  of  the 
unknown  viariables. 

d.  The  MOE  is  to  be  maximized  or  minimized. 

3.  Queueing  Theory 

a.  Queueing  Theory  is  applied  to  a  situation  where  "customers"  must  wait 
in  line  to  be  "served."  The  customers  can  be  people,  parts  to  be 
repaired,  or  data  to  be  stored  or  used  in  computations.  The  server  can 
be  a  person,  a  computer  terminal,  a  message  buffer,  or  a  machine. 

b.  Queueing  analysis  considers  the  number  of  servers,  the  serving 
discipline  (e.g.  first -come-first-serve),  number  of  customers  in  the 
queue,  service  rate,  and  customer  arrival  rate  and  pattern. 
Distributions  of  these  parameters  can  be  handled  analytically. 

c.  It  is  usually  assumed  that  the  random  variable  corresponding  to  the 
times  between  customer  arrival  in  the  queue  are  independent  and 
identically  distributed.  It  is  also  usually  assumed  that  the  service  times 
are  independent  and  identically  distributed  random  variables. 

d.  Common  outputs  desired  from  a  queueing  model  are  expected  number 
of  customers  (or  units)  in  the  queue  and  in  the  whole  system  under 
steady  state  conditions,  and  expected  time  spent  by  a  customer  (or  unit) 
in  the  queue  and  in  the  whole  system  under  steady  state  conditions. 


66 


sis 

'  v  % 


NWC  TP  6541 


TABLE  28.  Some  Analysis  Techniques.  (Continued) 


4.  Control  Theory 

a.  Control  Theory  is  applied  to  any  situation  where  it  is  desirable  to 
control  a  process  such  that  specified  states  are  maintained.  Controlling 
or  maintaining  the  speed  of  an  engine,  turntable,  or  cassette  tape  drive, 
the  altitude  of  an  aircraft,  or  water  flow  to  a  generator  all  require 
some  control. 

b.  Many  analytical  techniques  have  long  been  in  use  to  describe  control 
systems  and  estimate  their  stability  and  accuracy.  Use  of  differential 
equations,  Laplace  transform,  and  the  root-locus  method  have  been 
used  to  model  linear,  dynamic  systems. 

c.  The  human  operator  has  also  been  modeled  in  situations  where  he  must 
observe  a  process,  estimate  its  states,  and  generate  a  control  action. 
This  work  has  included  the  design  of  displays,  controllers,  and  control 
mechanism  characteristics  used  by  the  operator. 

5.  Game  Theory 

a.  Game  Theory  is  a  collection  of  mathematical  models  applied  to  the 
behavior  of  "players"  who  try  to  modify  the  state  of  a  system  to  attain 
specified  goals.  The  situation  being  modeled  is  one  oi  cooperation,  or 
one  of  competition  or  conflict.  Conflict  arises  when  two  or  more 
players  have  conflicting  goals  (such  as  shoot  the  other  player  down). 

b.  A  model,  or  game  is  defined  by  its  goals  and  its  operating  rules,  or 
strategy.  A  strategy  is  a  complete  description  of  how  a  player  will 
behave,  or  what  decisions  he  will  make  under  every  possible 
circumstance. 

c.  Game  Theory  provides  guidelines  for  rational  ben  vior  when  confronted 
with  tactical  or  strategic  decisions.  The  game  model  describes  all 
potential  payoffs,  and  identifies  actions  required  to  get  the  best 
possible  outcome  in  light  of  the  options  open  to  the  opponents  in  the 
game.  It  uses  optimization  procedures,  statistics,  probability,  and 
classical  decision  theory. 

d.  Game  Theory  is  used  to  assess  tactics  (strategy),  and  to  study 

decision-making,  but  has  not  been  used  much  in  determining  design 
requirements.  It  is  a  useful  tool,  however,  where  system 

characteristics  such  as  weapon  standoff  range  could  interact  strongly 
with  system  employment. 
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Developing  more  specific  recommendations  for  selection  of  analytic 
techniques  is  beyond  the  scope  of  this  report,  as  well  as  beyond  the  author.  As 

Martin30  advises, 

1)  look  at  the  real  world. 

2)  study  the  problem. 

3)  examine  the  desired  results  of  the  model. 

4)  examine  resources  available  and  time  schedule. 

5)  select  the  right  procedure  at  the  right  place  in  the  model. 

6)  document  the  rationale  for  the  selections  made. 

We  must  not  forget  the  consultants  on  our  analysis  team  (//3,  Table  10). 
They  can  certainly  advise  when  presented  with  the  information  assembled  thus 
far,  and  may  even  have  to  be  enlisted  to  do  the  work.  References  19  and  31  to 
40  will  provide  more  detailed  information  for  the  really  motivated. 


Rouse,  William  B.,  Systems  Engineering  Models  of  Human-Machine 
Interaction.  Elsevier  North  Holland,  Inc.,  New  York,  1980. 

32 

Allen,  Arnold  O.,  Probability,  Statistics,  and  Queueing  Theory,  Academic 
Press,  New  York,  1978. 

33 

Phillips,  D.  T.,  et  al,  Operations  Research,  Principles  and  Practice,  John 
Wiley  &  Sons,  Inc.,  New  York,  1976. 

34 

Buffa,  E.  S.  and  J.  S.  Dyer,  Management  Science/Operations  Research;  Model 


Formulation  and  Solution  Methods,  John  Wiley  &  Sons,  Inc.,  New  York,  1977. 


35 

Emshoff,  J.  R.  and  R.  L.  Sisson,  Design  and  Use  of  Computer  Simulation 
Models,  The  MacMillan  Co.,  New  York,  1970. 

36 

Boeing  Aerospace  Company,  Analyst's  Guide  for  the  Analysis  Sections  of 
MIL-H-46855.  by  Charles  W.  Geer,  Report  D180-19476-1,  30  June  1976,  Seattle, 
Washington. 

^  Davis,  Morton  D.,  Game  Theory:  A  Nontechnical  Introduction,  Basic  Books, 
Inc.,  New  York,  1970. 

38 

Blaquieve,  A.,  et  al,  Quantitative  and  Qualitative  Games,  Academic  Press, 
New  York,  1969. 

39 

Game  Theory  and  Its  Applications,  American  Mathematical  Society, 
Providence,  Rhode  Island,  1981. 

40 

Lucas,  W.  F.,  An  Overview  of  the  Mathematical  Theory  of  Games, 
Management  Science.  Vol.  18,  No.  5,  January  1972,  Part  2. 
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SUMMARY  OF  PROCEDURES 


The  steps  in  algorithm  development  that  have  been  described  above  are 
summarized  in  two  ways.  For  those  with  a  high  verbal  aptitude,  a  simple  list  of 
the  steps  is  given.  For  those  of  us  who  like  pictures,  Figures  16,  17,  and  18  show 
the  steps  in  block  diagram  form.  The  tables  of  instruction  given  throughout  the 
text  are  also  reproduced  so  that  they  will  be  available  in  one  place. 


START 


FIGURE  16.  Steps  in  Describing  What  the  System  Is  All  About. 
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PREPARATORY  DOCUMENTATION 

1.  Write  an  overview  description  of  the  system  and  the  way  it  will  be  used. 
(Table  29). 

2.  Have  the  overview  description  approved  by  other  principal  design  team 
members  and  by  the  project  sponsors. 

3.  Make  changes  in  overview  description  as  per  review  comments. 

4.  Have  the  overview  description  reviewed  by  users  of  a  related  or  similar 
system. 

5.  Incorporate  users'  review  comments  into  overview  description. 

6.  Write  a  detailed  system  description  including  alternate  configurations  being 
considered  and  showing  where  design  decisions  must  be  made.  (Table  30). 

7.  Have  the  detailed  system  description  reviewed  by  the  principal  design  team 
members. 

8.  Negotiate  and  incorporate  changes  in  the  detailed  system  description. 

9.  Write  a  mission  description  which  indicates  how  the  system  will  be  used  and 
what  functions  must  be  performed  to  insure  mission  success;  alternate  operating 
modes  should  be  included.  (Table  31). 

10.  Have  the  mission  description  reviewed  by  the  design  team  members. 

11.  Incorporate  changes  in  the  mission  description  resulting  from  the  review. 

12.  Write  a  description  of  the  operating  conditions,  including  the  environments 
of  both  the  hardware/software  and  the  human  operators. 

13.  Have  the  operating  conditions  reviewed  by  the  user  community. 

14.  Incorporate  changes  resulting  from  user  review. 

15.  Develop  Measures  of  Effectiveness  to  be  used  in  system  effectiveness 
studies  (Table  32). 

16.  Have  System  Description,  Missions,  Operating  Conditions,  and  MOEs 
reviewed  by  principal  design  team  members,  program  managers,  and  sponsors. 
This  is  the  whole  picture;  is  it  acceptable  by  everyone? 

17.  Incorporate  changes  resulting  from  the  reviews  in  (16). 
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BUILDING  THE  ALGORITHM 

18.  Select  functions  and  tasks,  system  components,  and  environmental  factors 
that  must  be  included  in  the  system  effectiveness  algorithm.  (Table  33). 

19.  Quantify  the  MOEs  that  were  developed  in  (15).  (Table  34). 

20.  Modify  the  MOEs  as  required  (Table  35). 

21.  If  MOE  modification  is  major,  have  the  new  MOEs  approved  by  same  group 
as  given  in  (16). 

22.  Draw  block  diagram(s)  showing  functions  to  be  performed  in  sequence  (in 
series  and/or  parallel  as  appropriate). 

23.  Draw  diagrams  showing  applicable  geometry  and/or  time  lines  as 
appropriate. 

24.  Relate  diagrams  in  (22)  and  (23)  to  system  components,  MOEs,  and  operating 
conditions. 

25.  Break  MOEs  developed  in  (19)  into  components  (if  appropriate)  that 
correspond  to  the  diagrams  built  under  (22)  and  (23). 

26.  Get  estimates  from  experts  or  consultants  of  the  availability  and  source  of 
performance  data  (use  something  like  Table  9)  for  each  component,  including  the 
human  operator(s). 

27.  Formulate  equations  or  get  data  (e.g.  in  the  form  of  look-up  tables)  that 
describe  system  component  performance  (including  the  human  operator)  in  terms 
of  the  MOE  components.  The  equations  or  data  should  characterize  (a)  the 
naturally  occurring  distributions  in  the  data  (Figure  14),  and  (b)  the  effects  of 
environmental  operating  conditions  (e.  g.  acceleration,  weather). 

28.  Develop  procedures  for  combining  the  MOE  components  to  produce  the 
desired  MOEs.  This  step  entails  selecting  mathematical  tools  and/or  models 
(Tables  27  and  28)  appropriate  to  the  particular  problem.  Consultation  with,  or 
active  participation  by  specialists  (e.g.,  mathematicians)  will  probably  be 
required.  The  resultant  MOE  should  be  either  of  the  Stochastic  or 
Deterministic/Probabilistic  type  (Table  27)  such  that  any  desired  level  of  system 
effectiveness  (Figure  15)  can  be  determined. 

29.  Verify  the  algorithm's  internal  consistency  with  several  special  sets  of 
inputs  for  which  the  outputs  can  be  predicted.  It  is  important  to  be  able  to 
explain  the  algorithm's  outputs  in  terms  directly  related  to  the  real  world: 
system  characteristics,  operator  performance,  and  operating  conditions. 
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1.  Define  and  Document  the  Objective. 

2.  Define  and  Document  Special  Requirements. 

3.  Define  and  Document  the  System  (broken  into  Major  System  Components). 

4.  Define  and  Document  the  Operator(s)  Characteristics. 

5.  Identify  and  Document  the  Operating  Times,  Places,  and  Environmental 
Conditions  (see  Table  3). 

6.  Identify  and  Document  Other  Factors  or  Entities  Affecting  System 
Operation,  (See  Table  3). 

7.  Describe  and  Document  the  Operating  Concepts. 

a.  How  will  the  system  be  used? 

b.  What  role  do  major  system  components  play? 

8.  Determine  General  System  Measures  of  Effectiveness. 


TABLE  30.  Steps  in  System  Description. 


1.  Use  information  describing  the  system  that  already  exists. 

2.  Include  only  those  components  that  are  thought  to  relate  to  overall  system 
effectiveness.  Keep  this  first  description  general;  don't  describe  the 
detailed  parts  of  a  component. 

3.  Identify  design  decisions  yet  to  be  made  on  specific  components.  That  is, 
indicate  which  components  are  still  selectable,  and  indicate  the  range  in 
characteristics  that  is  still  possible. 

4.  Include  alternative  configurations  if  a  specific  one  has  not  yet  been  adopted. 

5.  Have  the  system  description  reviewed  by  the  design  team. 
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TABLE  31.  Notes  on  Mission  Description. 

1.  List  and  briefly  describe  functions  that  must  be  performed  to  accomplish 
the  mission. 

2.  List  and  briefly  describe  the  major  tasks  required  in  each  function.  A 
function  and  task  allocation  is  implied  here.  It  should  have  been  done 
elsewhere  as  part  of  the  human  factors  work  on  the  program.  It  may  have 
been  done  implicitly  by  the  design  engineers. 

3.  Do  the  above  (1  ic  2)  for  each  configuration  and  mission,  or  show  branches  in 
the  appropriate  mission  segments  to  include  the  alternatives. 

4.  Construct  mission  profiles  for  illustrating  the  concepts  to  reviewers.  Those 
for  aircraft  missions  sometimes  show  the  flight  path  plotted  by  altitude  and 
ground-track  with  functions  or  tasks  indicated.  Others  can  show  a  timeline, 
with  the  function  or  tasks  shown  in  the  proper  sequence. 

5.  Note  assumptions,  alternatives,  questions,  or  issues  that  come  up  in  doing 
this  work.  This  side  information  may  prove  helpful  later,  or  must  be 
resolved  if  controversial.  A  systematic  procedure  should  be  established  to 
keep  a  record  of  changes  in  the  missions  as  the  development  of  the  system 
progresses. 

6.  Have  completed  missions  reviewed  by  design  team  and  users. 


MOEs  should: 


1.  Be  required  in  some  decision  process  (e.g.,  component  selection). 

2.  Include  aspects  of  the  physical  environment  that  affect  operator  and 
system  performance. 

3.  Use  variables  that  are  readily  measurable  in  the  real  world,  and/or  for 
which  there  is  a  data  base. 


Some  Guidelines  for  Developing  MOEs 

1.  List  important  mission  features  so  that  the  MOEs  have  a  better  chance 
of  reflecting  the  way  a  mission  must  be  conducted  in  order  to  be  effective. 

2.  Develop  a  list  of  conceivable  MOEs  for  the  missions;  this  brainstorming 
session  should  be  conducted  without  constraints  (list  all  possibilities). 

3.  Reduce  the  list  by  discarding  duplication  and  MOEs  that  are  not  in  some 
way  related  to  the  mission  objectives. 

4.  Write  brief  discussion  of  each  of  the  MOE  and  tabulate  some  of  the 
general  characteristics  of  each  MOE. 

5.  Categorize  the  MOEs  into  groups  of  similar  measures. 

6.  Select  the  best  MOEs  in  each  group  using  a  procedure  to  evaluate  those 
that  are  strong  or  weak,  or  alike  or  similar. 

7.  Point  the  selected  MOEs  to  the  next  higher  level  of  objectives:  i.e., 
insure  that  the  MOEs  are  so  constructed  that  they  can  serve  as  performance 
indicators  to  the  next  higher  level. 

8.  Express  the  MOEs  in  standard  notation  of  physics,  engineering,  and 
mathematics  (i.e.,time  required,  distance  covered,  percent  defects  identified). 
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TABLE  33.  Steps  in  Selecting  Items  to  be  Modeled. 


.*■  . 
v;-: 

1. 

Augment  the  list  of  functions  and  tasks  with  comments,  remarks,  or 
observations  to  provide  a  more  complete  description  of  the  items. 

a 

u 

2. 

Estimate  the  strength  of  the  effect  on  the  MOEs  of  each  of  the  tasks. 
Include  secondary  effects. 

1 

3. 

Indicate  which  tasks  are  affected  by  the  operating  conditions  listed  earlier. 
Pair  up  specific  tasks  with  the  specific  operating  conditions. 

4. 

Make  a  preliminary  decision  as  to  what  parts  of  the  system  and  system 
operation  should  be  modeled. 

V 

5. 

Document  the  above. 

~T| 

i' 


TABLE  34.  Steps  in  MOE  Quantification. 

1.  Start  with  the  MOEs  listed  and  agreed  upon  earlier. 

2.  Break  the  MOEs  into  components,  if  possible,  such  that  each  can  be  related 
to  the  functions  and  tasks  listed  earlier.  Include  only  those  items  to  be 
modeled. 

3.  If  the  MOEs  cannot  be  simply  broken  into  components  as  in  (2),  develop  the 
special  definitions  required  to  translate  the  system  (or  subsystem) 
performance  into  the  appropriate  MOEs.  These  definitions  can  usually  take 
the  form  of  mathematical  statements.  Document  the  assumptions  made  in 
this  process. 

4.  Indicate  which  equipment  component(s)  and/or  human  operator(s)  are  related 
to  each  MOE  component.  (The  MOE  component  is  really  a  measure  of 
performance). 

5.  Indicate  (qualitatively)  how  design  decisions  might  be  affected  by  the  MOEs. 
(This  item  is  an  "extra"  that  reminds  us  why  we  are  doing  the  anlaysis  in  the 
first  place.) 

6.  Verify  that  the  MOE  components  are  compatible  with  any  lower-level  MOEs 
that  may  be  used  in  analysis  or  testing. 

7.  Show  how  the  MOE  components,  or  special  MOE  definitions,  can  be 
"reassembled"  to  form  the  top-level  MOE  first  described.  A  block  diagram 
may  be  useful  in  this  process. 

8.  Have  MOEs  that  are  formulated  in  item  (3)  above  reviewed  by  any  other 
analysts  on  the  program  and  principal  design  team  members. 


V*  ■v  <■ 


■■rV  <v' 


< 


NWC  TP  6541 

TABLE  35.  Advantages  and  Disadvantages  in  Modifying  MOEs. 


Advantages 

1.  Modification  may  be  required  as  indicated  by  a  more  detailed  look  at  the 
system  and  its  operation.  The  original  MOEs  may  simply  not  fit  the 
situation. 

2.  Modification  may  better  match  the  MOE  components  to  the  equipment  and 
operator  performance. 

3.  Modification  may  save  some  work  (e.g.,  specifically  describing  a  mission  or 
scenario). 


Disadvantages 

1.  Modification  may  not  be  acceptable  to  the  sponsor  (customer).  It  simply 
cannot  be  done  in  this  case. 

2.  Modification  will  require  another  iteration  in  all  work  done  to  this  point  to 
insure  compatibility. 


Requirement 

1.  The  modified  MOEs  must  still  serve  the  desired  purposes  (e.g.  provide 
information  for  use  in  making  design  decisions). 

2.  If  the  modifications  to  the  MOEs  are  simplifications,  the  new  MOEs  should 
be  useful  in  constructing  the  original  MOEs  if  more  time  (resources)  become 
available  for  analysis. 


/,  A 
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IMPLICATIONS  FOR  HUMAN  FACTORS  EXPERIMENTS 


The  problem  of  finding  applicable  human  performance  data  for  use  in 
analysis  has  been  discussed  in  the  main  text  as  well  as  in  Appendix  C.  Some  of 
the  inadequacies  in  the  data  are  given  below. 

1.  Most  experiments  include  mly  one  or  two  variables,  whereas  the  real 
world  situation  may  have  several  variables  which  can  affect  performance. 

2.  Sometimes  the  experiments  do  not  include  the  variables  of  interest  at 
all,  even  though  the  general  situation  and  performance  measures  may  be 
applicable. 

3.  The  range  of  the  variables  (e.g.,  65  to  80°F)  may  not  be  the  range 
estimated  in  the  employment  of  the  system. 

4.  The  subjects  in  the  experiment  may  not  be  taken  from  the  population 
that  would  be  using  the  system  (e.g.,  female  schoolteachers  versus  male  fighter 
pilots). 

5.  The  experiment  may  not  have  been  conducted  in  the  context  of  system 
operation. 

6.  All  of  the  data  (e.  g..  Figure  14)  are  not  reported.  Only  means  or 
analysis  of  variance  tables,  or  the  like,  are  given. 

41  42  43 

Simon  ’  and  Williges  provide  considerable  discussion  on  the  short¬ 
comings  of  human  factors  experiments  and,  more  importantly,  recommend 
experimental  designs  that  will  improve  the  results.  The  designs  allow  an 
experimenter  to  include  a  large  number  of  variables  in  a  series  of  experiments. 

Experimenters  could  improve  the  applicability  of  their  results  by  including 
conditions  of  no  immediate  interest.  For  example,  a  wider  range  of  values  for 
selected  parameters  could  be  used  than  called  for  in  the  anticipated  application 
without  compromising  the  primary  objective  of  the  experiment. 


Simon,  Charles  W.,  New  Research  Paradigm  For  Applied  Experimental 
Psychology:  A  System  Approach,  Canyon  Research  Group  Report  CWS-04-77, 
October  1977,  Westlake  Village,  Calif.  (U5AF  Contract  No.  F44620-76-C-0008.) 
U7 

Simon,  Charles  W.,  Design,  Analysis,  and  Interpretation  of  Screening  Studies 
for  Human  Factors  Engineering  Research,  Canyon  Research  Group  Report 
CW5-03-77,  September  1977,  Westlake  Village,  Calif.  (USAF  Contract 
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It  is  common  that  subjects  for  experiments  are  selected  ("drafted"  is  often 
more  appropriate)  from  beginning  psychology  classes  or  from  the  experimenter's 
colleagues.  A  special  effort  should  be  made  to  (1)  specify  the  most  likely  user 
population,  and  (2)  select  subjects  from  this  population. 

Conducting  an  experiment  in  the  context  of  system  operation  can  be  one  of 
the  most  unattainable  corrections  to  the  above  deficiency  list,  primarily  because 
of  economics.  The  limited  availability  and/or  high  cost  of  operating  an 
appropriate  simulator  may  preclude  its  use.  The  experimenter  must  often  settle 
for  less.  There  is  no  solution  to  this  problem,  other  than  to  be  aware  of  the 
spectrum  of  data  sources  (Table  6),  try  to  get  the  "best,"  and  interpret  the 
resulting  data  accordingly. 

And  for  heaven's  sakes,  report  all  of  the  data!  It  won't  cost  that  much  more. 
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Appendix  A 

ADDITIONAL  BACKGROUND  MATERIAL 


SYSTEMS 

Several  definitions  of  systems  have  been  given  in  the  human  factors 
literature.  Examples  are  given  below. 

A  system  is  "a  number  of  parts  which  are  connected  together 
in  order  to  transform  a  given  set  of  inputs  into  a  given  set  of 
outputs.”  (Jones,  reference  16). 

Jones  also  gives  11  categories  for  classifying  systems  according  to  mode  of 
operation  and  physical  nature  of  their  components  and  couplings. 

"A  man-machine  system  can  be  viewed  as  any  organized 
group  of  activities,  involving  men  and  machines,  directed 
towards  the  solution  of  a  given  problem  or  set  of  problems  and 
operating  within  the  constraints  of  a  given  environment." 

(Corkindale^). 

"A  system  is  a  set  of  interacting  components. 

Components  are  physical  entities  such  as  human  operators  or 
electric  motors  or  planets.  Interaction  implies  that  they  are 
capable  of  exchanging  energy  and  information.  Assuming  that 
the  designer  is  behaving  rationally,  every  man-made  system 
must  have  a  purpose:  That  is,  a  set  of  objectives." 

(Singleton^) 

"A  system  is  an  organization  in  which  the  individual 
elements  work  together  purposefully  to  produce  an  output 
which  the  individual  element  cannot  produce  by  itself." 

(Meister^) 

"A  man-machine  system  is  essentially  a  concept  based  on 
certain  assumptions  (to  be  described  later);  it  is  an 
abstraction,  not  a  physical  configuration  or  a  type  of 
organization."  (Meister^) 


Corkindale,  K.  G.,  "Man-Machine  Allocation  in  Military  Systems," 
ERGONOMICS  March  1967,  Vol.  10,  No.  2,  pp.  161-166. 
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Meister,  David,  "A  Systematic  Approach  to  Human  Factors  Measurement," 
Navy  Personnel  Research  and  Development  Center,  October  1978. 
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"Consequently,  the  concept  of  a  "system"  is  an  abstract, 
devised,  synthetic  entity.  ...  it  is  a  "system"  only  because 
someone  views  it  from  a  given  point  of  reference.  He  sees  an 
organization  or  an  integration  of  forces  or  events  for  which  he 
can  define  a  set  of  boundaries."  (Chase*®) 


SYSTEMS  DESIGN  PROCEDURES 

Procedures  to  be  followed  in  systems  design  and  development  have  been 
diagrammed  by  several  authors.  Figures  A-l  through  A-6  illustrate  the  various 
concepts  and  procedures;  they  illustrate  that  there  is  more  than  one  way  to  skin 
a  cat,  as  my  grandfather  used  to  say.  Figure  A-7  lists  some  of  the  parameters 
included  in  the  analysis  procedures. 


system 

REQUIREMENTS 


DETERMINATION  OF 
SYSTEM/SUBSYSTEM 
FUNCTIONS 


DESIGN  ALTERNATIVE 


___  BEHAVIOURAL-! 

|ENGineering'"-'-.^nalyses  j 

I _ ANALYSES 


DESIGN  ALTERNATIVE 


DESIGN  ALTERNATIVE 


DESIGN  ALTERNATIVE  I  PROCESS  I 


PRELIMINARY 

SELECTED 

CONFIGURATION 


CONFIGURATION  | 
TESTING  s 


FINAL 

CONFIGURATION 


,  CONFIGURATION  . 
"1  REVISION  r 

I _ I 


FIGURE  A-l.  The  System  Development  Process  (from  Meister  5). 
(Note:  Broken  boxes  represent  analyses;  unbroken  boxes,  inputs/outputs) 


I? 


FIGURE  A-2.  General  Steps  in  Systems  Design  (from  Chase,  reference  10), 
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FIGURE  A-4.  Typical  Procedure  for  Determining  System 
Conception  and  Structure  (Ilyama  ). 

^  Iiyama,  Yuji,  "Systems-Ergonomic  Approaches  to  Design  and  Operation  of 
Today's  Railroads,"  Human  Factors  1980,  Vol  22,  No.  1,  pp.  15-24. 
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PROBLEM  DEFINITION 


•  ENVIRONMENTAL  FACTORS 

•  STATE-OF-TH  E-ART 
•TECHNOLOGICAL  FACTORS 
•HUMAN  FACTORS 

•PROGRAMMED  CAPABILITIES 

•  CONSTRAINTS 

•  ORGANIZATIONAL 

FACTORS 

•  SOCIO-POLITICAL 

FACTORS 


KNOWLEDGE  OF 
TECHNOLOGY 


HUMAN 

ENGINEERING 

KNOWLEDGE 


CREATIVITY 


CONCEPT  FORMATION 


GENERATION  OF 
ALTERNATIVES 


FUNCTION  ALLOCATION 


1 

tQUIPMtNT 

MAINTENANCE  & 
SUPPORT  CONCEPT 


OPERATIONAL 

CONCEPT 


MISSION  ANALYSIS 

•  FUNCTIONS 

•  MODELS 

•  SCENARIOS 

•  REQUIREMENTS 


COST  DATA 


FIGURE  A- 5.  The  General  Process  of  Concept  Formation 

kl 

Related  to  Other  System  Analysis  Phases. 


*  Schneider,  R.  H.,  et  al,  System  Analysis  Guide,  Dunlap  and  Associates,  Santa 
Monica,  Calif.  (Report  No.  40-WA3-1,  Contract  No.  N123(60530)  53431A  with 
the  Naval  Weapons  Center,  China  Lake,  publication  UNCLASSIFIED),  October 
1966. 
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CODE: 

zTD  and 

- ITERATIVE  PROCESS 

FIGURE  A-6.  Principal  Tasks  Required  for  Evaluation 
of  System  Effectiveness  (from  Hermann21). 
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1.  OPERATIONAL  MISSION-OBJECTIVES 

2.  DESCRIPTION  OF  CONDITIONS  FOR 
DEPLOYMENT 

3.  CONCEPT  OF  OPERATIONS  AND  SUPPORT 

4.  OPERATIONAL  PERFORMANCE  REQUIREMENTS 

6.  SYSTEM  EFFECTIVENESS  CRITERIA 

5.  DATE  SYSTEM  IS  TO  BE  AVAILABLE 

7.  COST  LIMITATIONS 

8  DEVELOPMENT  PROGRAM  CONSTRAINTS 

S.  POLITICAL  CONSTRAINTS 

10.  DESCRIPTION  OF  THE  TECHNICAL 

APPROACH  WHICH  IS  TO  BE  VALIDATED. 
INCLUDING  ANY  OFF-THE-SHELF' 

ITEMS  WHICH  ARE  TO  BE  USED  OR  TO 
WHICH  THE  SYSTEM  IS  TO  BE  MATED 


SYSTEM  FUNCTIONAL  REQUIREMENTS 


PROGRESSIVE  ANALYSIS  OF  SYSTEM 
REQUIREMENTS  AND  CONTRAINTS  TO: 

1.  FORMULATE  TOTAL  SYSTEM  MISSION  AND 
FUNCTIONAL  REQUIREMENTS  (FUNCTIONAL 
FLOW  DIAGRAMS) 

2.  APPORTION  PERFORMANCE  REQUIREMENTS 
AND  RELIABILITY  VALUES  TO  SUBSYSTEMS 
(SYSTEM  EFFECTIVENESS  MODEL) 

3.  DESCRIBE  FUNCTIONAL  DESIGN  CHARACTERISTICS 
TO  ACHIEVE  AN  INTEGRATED  SYSTEM 

DESIGN  (FUNCTIONAL  SCHEMATIC) 

4.  ESTABLISH  CRITERIA  AND  STANDARDS  FOR 
EVALUATION  OF  SYSTEM  END  ITEM 
PERFORMANCE  (TIME-LINES) 

5.  INTERRELATE  SYSTEM  AND  FUNCTIONAL 
REQUIREMENTS  WITH  PERFORMANCE  ALLOCATIONS 
AND  ANALYZE  OMISSIONS  (REQUIREMENTS 
MATRIX).  THESE  WILL  INVOLVE: 

A.  PRIME  MISSION  EQUIPMENT 

B.  SUPPORT  EQUIPMENT 

C.  FACILITIES 

D.  PROCEDURES 

E.  PERSONNEL  TRAINING 

F.  LOGISTICS  SUPPORT 


SYSTEM  DESIGN 


SYSTEM  TRADE  OFF  STUDIES 


DERIVE  A  COHERENT  SYSTEM  DESIGN 
TO  PRODUCE  A  DEFINED  SET  OF 
OPTIMUM  OUTPUTS  FROM  GIVEN  INPUTS 
WITH  RESPECT  TO  TIME.  COST. 

AND  PERFORMANCE  MEASURES  OF 
EFFECTIVENESS: 

1.  ALTERNATIVE  OESIGN  APPROACHES 
MAY  BE  SPECIFIED  FOR  DEFERRED 
EVALUATION  AND  FINAL  CHOICE 
DURING  SYSTEM  ACQUISITION 

2.  ACQUISITION  MAY  BE  DELAYED 
PENDING  DEVELOPMENT  OF  KEY 
COMPONENTS 

3.  DESIGN  OBJECTIVES  OR  PERFORMANCE 
REQUIREMENTS  MAY  BE  COMPROMISED 
IF  THE  URGENCY  TO  ACQUIRE  SOME 
DEGREE  OF  THE  SPECIFIED  OPERATIONAL 
CAPABILITY  SO  DICTATES 


CONCERNED  WITH  ACHIEVING  OVERALL  SYSTEM 
OPERATIONAL  CAPABILITY  BY 
1  DEVELOPING  A  DESCRIPTIVE  SYSTEM 
MODEL,  OR  ALTERNATIVE  MODELS  TO 
INTEGRATE  INPUTS.  OUTPUTS  AND 
SYSTEM  STATES  IN  A  REAL  TIME 
OPERATIONAL  ENVIRONMENT 
2.  EVALUATING  ALTERNATIVES  AND 

VARIATIONS  IN  DESIGN  AND  SELECTING 
BEST-FIT'  SYNTHESIS  OF  SOLUTIONS 
FOR  MANAGEMENT  CONSIDERATIONS 
IN  RELATION  TO  PERFORMANCE. 

A.  DETERMINING  HOW  THE  BEST 
ACHIEVABLE  DESIGN  SOLUTION 

EXCEEDS  OR  FALLS  SHORT  OF  REQUIREMENTS 

B.  IDENTIFYING  WHEN  DESIGN  AND 
DEVELOPMENT  OF  A  SYSTEM  ELEMENT 
REQUIRES  ACHIEVEMENT  OF  NEW 
TECHNOLOGY  GOALS  AND  ESTIMATING 
probability  of  success  within 

SYSTEM  DEVELOPMENT  CONSTRAINTS 
(  HIGH  RISK  AREAS') 

C.  IDENTIFYING  PROBLEMS  REQUIRING 
SPECIAL  MANAGEMENT  ATTENTION  AND 
MAXIMUM  VISIBILITY  FOR  MONITORING 
PROGRESS  IN  DEVELOPMENT 


FIGURE  A-7.  Basic  System  Engineering  Parameters 
As  Adapted  from  Chase 
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APPENDIX  B 

MEASURE  OF  EFFECTIVENESS 


There  are  a  number  of  sources  of  information  on  measures  of  effectiveness, 
but  a  primer  or  textbook  on  the  subject  has  not  been  located.  Ideas  about 
properties  of  good  MOEs,  and  the  procedures  for  developing  MOEs  for  a 
particular  study  can  be  obtained  from  selected  quotes  from  the  literature. 

22 

Criteria  for  MOE  (taken  from  Anderson,  et  al  ) 

Measures  of  effectiveness  have  to  be  developed  for  each  new  study  and  test 
because  no  one  set  of  MOE  has  been  forwarded  that  fits  all  situations.  Two  of 
the  criteria  for  selecting  final  MOE  from  a  list  of  considered  measures  are  given 
below. 


1.  The  first  criterion  is  that  a  MOE  express  the  extent  to  which  a  system 
meets  the  best  possible  performance.  A  MOE  may  express  system  performance 
as  a  proportion  of  maximum  performance.  For  example,  percent  of  targets  hit 
and  probability  of  hit  are  MOE,  and  firing  rate  is  a  measure  of  performance  that 
can  be  converted  into  a  MOE  by  dividing  it  by  maximum  required  (or  desired) 
firing  rate.  Obviously,  making  the  best  required  performance  the  denominator  of 
the  measure  means  that  MOE  can  only  be  developed  in  keeping  with  the 
objectives  of  a  system. 

2.  The  second  criterion  for  MOE  is  that  they  should  be  consistent  in 
quantities  and  units.  This  facilitates  intra-  and  inter-  system  comparisons.  This 
criterion  requires  that  the  analyst  consider  how  the  numbers  expressing  the  MOE 
are  to  be  manipulated  mathematically  in  order  to  derive  conclusions  concerning 
the  effectiveness  of  the  systems  being  evaluated. 

Figure  B-l  illustrates  that  one  "effectiveness"  results  from  a  fixed  scenario. 
In  practice,  however,  if  the  effectiveness  is  not  as  high  as  desired,  the  operating 
conditions  can  be  changed  and  the  operators  can  be  trained  to  improve  the 
situation  (Figure  B-2).  Figure  B-3  illustrates  that  an  effectiveness  analysis  is  an 
iterative  procedure,  and  can  affect  the  system  design,  tactics  selection,  and 
training  program  for  a  system  under  development. 

This  iterative  nature  in  the  design  and  use  of  a  system  makes  it  difficult  to 
come  up  with  a  "final"  anlaysis  or  model.  The  analysis  should  be  structured  at 
the  outset  so  that  changes  can  be  made  without  too  much  difficulty  (e.g.  a 
modular  design). 
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FIGURE  B-l.  Open-Loop  System  Effectiveness  Diagram. 


FIGURE  B-2.  Existing  System  Effectiveness  Diagram 
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SYSTEM  DESIGN 
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APPENDIX  C 

AVAILABILITY  OF  PERFORMANCE  DATA 


For  many  years,  analysts  have  had  difficulty  in  finding  appropriate  operator 
performance  data  for  use  in  their  studies.  Some  quotes  from  the  literature  will 
give  the  reader  an  idea  of  the  difficulties. 

"At  first  g.'ance,  the  massive  amount  of  human 
performance  data  existing  in  psychological  journals  and 
technical  reports  might  seem  more  than  sufficient  for 
application  to  man-machine  models  by  the  naive  observer. 
Several  attempts  have  been  made  to  extract  relevant  data 
from  that  literature.  However,  none  of  the  resulting  "data 
stores"  appears  so  far  to  have  any  substantial  degree  of 
validity  or  utility,  and  there  is  considerable  doubt  that  the 
literature  extraction  process  will  ever  produce  data 

appropriate  to  modeling  purposes."  (from  Smith,  et  ai,^) 

Or,  Meister^: 

"One  wonders  why  there  is  such  a  lack  of  available 
ergonomic  data,  considering  the  large  number  of  experimental 
studies  turned  out  annually.  Attempts  have  beeri  made  to 
translate  data  from  the  general  behavioral  literature  into 
ergonomic  equivalents  (Meister  and  Mills  1971'  but  the  diffi¬ 
culty  appears  to  be  that  the  tasks  performed  and  the  stimuli 
presented  in  general  psychological  studies  bear  little  resemb¬ 
lance  to  the  real  world  situations  faced  by  the  systems 
ergonomist." 

DeGreene*7  also  recognizes  the  problem: 

"The  problem  of  mismatch  between  operational  data 
needs  and  research  outputs  has  previously  been  discussed  in 
the  literature.  Variously  emphasized  have  been  the  lack  of 
transferability  of  laboratory  results  to  the  real  world,  frag¬ 
mentation  of  and  poor  communications  between  the  scientific 


Smith,  Russell  L.,  et  al,  "The  Status  of  Maintainability  Models:  A  Critical 
Review,"  Human  Factors  1970,  12(3),  pp.  271-283. 
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displines,  preoccupation  with  outmoded  concepts  of  'basic' 
research,  continued  dependence  on  obsolete  paradigms 
stemming  from  earlier  eras  of  scientific  research,  the  nature 
of  the  basic  unit  of  analysis,  lack  of  a  systems  approach  and 
lack  of  integrating  theory." 

49 

As  does  Simon  : 

"While  human  factors  practitioners  have  made  significant 
contributions  toward  easing  the  job  of  the  human  operator  and 
making  system  performance  more  effective,  the  contributions 
of  the  human  factors  scientists  —  the  experimenter  --  have 
been  modest.  Today,  one  has  to  search  diligently  among  piles 
of  published  papers  to  find  among  the  trivia  and  the  isolated 
facts,  data  that  is  sufficiently  generalizable  to  answer 
questions  concerning  the  design  of  future  systems  and  to  do  so 
quantitatively." 

"So  what  is  a  body  to  do"?  as  the  grandmother  in  the  soap  opera  would  ask. 

Blanchard5®  illustrates  how  data  availability  was  estimated  in  one  case,  and 

discusses  the  lack  of  data  and  the  possibility  of  establishing  a  store  of  data. 

Survey  respondents  had  the  following  opinions: 

"Utility  of  Data  Currently  Available 

"Perceptions  of  survey  respondents  were  also  gained  on 
the  utility  of  three  sources  of  data  which  are  generally 
available  and  in  use.  Those  sources  were:  (1)  experimental 
literature;  (2)  guides  and  manuals;  and  (3)  fleet  exercise  data. 

"The  available  experimental  literature  was  perceived  to 
represent  a  highly  limited  source  of  useful  data.  Several 
respondents  indicated  that  the  time  and  effort  spent  searching 
the  literature  for  data  seemed  to  be  seldom  worth  the  effort. 

Most  respondents  felt  that  the  experimental  literature  was 
essentially  non-applicable  to  applied  work  on  Navy  systems. 
Basically,  that  was  due  to  questions  as  to  the  generalizabiiity 
of  the  data  and  difficulty  in  determining  the  experimental 
circumstances  surrounding  data  collection  and  reporting. 


Simon,  Charles  W.,  Analysis  of  Human  Factors  Engineering  Experiments: 
Characteristics,  Results,  and  Applications,  Canyon  Research  Group,  Technical 


Report  CWS-02-76,  August  1976,  Westlake  Village,  Calif. 

5®  Blanchard,  R.  E.,  "Human  Performance  and  Personnel  Resource  Data  Store 
Design  Guidelines,"  Human  Factors  1975,  17(1)  pp.  25-34. 
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"  Human  engineering  guides  and  manuals  were  considered  to 
have  only  limited  utility  in  serious  system  design  efforts  where 
the  practitioner  is  attempting  to  obtain  quantified  estimates 
of  human  performance  relative  to  various  hardware  design  and 
environmental  parameters. 

"Fleet  exercise  data  were  not  considered  to  be  a  highly 
useful  data  source  since  the  data  were  felt  to  be  essentially 
undependable.  The  attempt  has  been  made  on  numerous 
occasions  to  use  observers  to  obtain  real-time  measures  of 
performance.  However,  the  data  are  felt  to  be  vulnerable  to 
errors  made  by  the  observers.  Consequently,  such  data  are 
viewed  with  suspicion  by  most  users." 
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